[==========] Running 25 tests from 3 test suites.
[----------] Global test environment set-up.
[----------] 21 tests from TxnParticipantITest
[ RUN ] TxnParticipantITest.TestReplicateParticipantOps
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251026 03:07:44.863139 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:41369
I20251026 03:07:44.863458 31233 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 03:07:44.863610 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:44.867326 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.867336 31239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.867326 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:44.867820 31233 server_base.cc:1047] running on GCE node
I20251026 03:07:44.868003 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:44.868043 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:44.868065 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448064868065 us; error 0 us; skew 500 ppm
I20251026 03:07:44.869045 31233 webserver.cc:492] Webserver started at http://127.30.128.126:45721/ using document root <none> and password file <none>
I20251026 03:07:44.869200 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:44.869242 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:44.869350 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:44.870374 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "f72a49c0207c408ca32478cebd4763f7"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.871555 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 03:07:44.872064 31246 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.872216 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 03:07:44.872292 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "f72a49c0207c408ca32478cebd4763f7"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.872350 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:44.881309 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:44.881551 31233 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 03:07:44.881654 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:44.885561 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:41369
I20251026 03:07:44.885587 31298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:41369 every 8 connection(s)
I20251026 03:07:44.886220 31299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:44.888527 31299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7: Bootstrap starting.
I20251026 03:07:44.889273 31299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:44.889475 31299 log.cc:826] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7: Log is configured to *not* fsync() on all Append() calls
I20251026 03:07:44.890084 31299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7: No bootstrap required, opened a new log
I20251026 03:07:44.891151 31299 raft_consensus.cc:359] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72a49c0207c408ca32478cebd4763f7" member_type: VOTER }
I20251026 03:07:44.891234 31299 raft_consensus.cc:385] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:44.891251 31299 raft_consensus.cc:740] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72a49c0207c408ca32478cebd4763f7, State: Initialized, Role: FOLLOWER
I20251026 03:07:44.891337 31299 consensus_queue.cc:260] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72a49c0207c408ca32478cebd4763f7" member_type: VOTER }
I20251026 03:07:44.891393 31299 raft_consensus.cc:399] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:07:44.891419 31299 raft_consensus.cc:493] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:07:44.891446 31299 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:44.891863 31299 raft_consensus.cc:515] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72a49c0207c408ca32478cebd4763f7" member_type: VOTER }
I20251026 03:07:44.891954 31299 leader_election.cc:304] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f72a49c0207c408ca32478cebd4763f7; no voters:
I20251026 03:07:44.892071 31299 leader_election.cc:290] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:07:44.892132 31302 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:07:44.892277 31299 sys_catalog.cc:565] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:07:44.892294 31302 raft_consensus.cc:697] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 1 LEADER]: Becoming Leader. State: Replica: f72a49c0207c408ca32478cebd4763f7, State: Running, Role: LEADER
I20251026 03:07:44.892366 31302 consensus_queue.cc:237] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72a49c0207c408ca32478cebd4763f7" member_type: VOTER }
I20251026 03:07:44.892681 31304 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f72a49c0207c408ca32478cebd4763f7. Latest consensus state: current_term: 1 leader_uuid: "f72a49c0207c408ca32478cebd4763f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72a49c0207c408ca32478cebd4763f7" member_type: VOTER } }
I20251026 03:07:44.892709 31303 sys_catalog.cc:455] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f72a49c0207c408ca32478cebd4763f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f72a49c0207c408ca32478cebd4763f7" member_type: VOTER } }
I20251026 03:07:44.892820 31304 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [sys.catalog]: This master's current role is: LEADER
I20251026 03:07:44.892831 31303 sys_catalog.cc:458] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [sys.catalog]: This master's current role is: LEADER
I20251026 03:07:44.893136 31309 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:07:44.893585 31309 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:07:44.893878 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:07:44.894927 31309 catalog_manager.cc:1357] Generated new cluster ID: f7bba8d1c85440b3bbe12e0b39916151
I20251026 03:07:44.894973 31309 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:07:44.905258 31309 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:07:44.905745 31309 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:07:44.910370 31309 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7: Generated new TSK 0
I20251026 03:07:44.910519 31309 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:07:44.925902 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:44.927239 31321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.927330 31324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.927516 31322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:44.927552 31233 server_base.cc:1047] running on GCE node
I20251026 03:07:44.927735 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:44.927771 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:44.927789 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448064927790 us; error 0 us; skew 500 ppm
I20251026 03:07:44.928428 31233 webserver.cc:492] Webserver started at http://127.30.128.65:42387/ using document root <none> and password file <none>
I20251026 03:07:44.928517 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:44.928567 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:44.928614 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:44.928907 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "7ab40f51249245bcacf1639206e8c30b"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.930009 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:44.930554 31329 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.930691 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.930733 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "7ab40f51249245bcacf1639206e8c30b"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.930783 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:44.946705 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:44.946962 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:44.947177 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:07:44.947505 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:07:44.947539 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.947572 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:07:44.947611 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.951391 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:46701
I20251026 03:07:44.951445 31392 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:46701 every 8 connection(s)
I20251026 03:07:44.951803 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:44.953110 31398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.953106 31397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:44.953377 31233 server_base.cc:1047] running on GCE node
W20251026 03:07:44.953873 31400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:44.954089 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:44.954129 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:44.954149 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448064954148 us; error 0 us; skew 500 ppm
I20251026 03:07:44.954787 31233 webserver.cc:492] Webserver started at http://127.30.128.66:38965/ using document root <none> and password file <none>
I20251026 03:07:44.954878 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:44.954924 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:44.954972 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:44.955248 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "9ccf56e5861b4cba8562031d12fcb535"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.956696 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:07:44.957604 31393 heartbeater.cc:344] Connected to a master server at 127.30.128.126:41369
I20251026 03:07:44.957682 31393 heartbeater.cc:461] Registering TS with master...
I20251026 03:07:44.957873 31393 heartbeater.cc:507] Master 127.30.128.126:41369 requested a full tablet report, sending...
I20251026 03:07:44.957466 31405 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.958125 31233 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251026 03:07:44.958166 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "9ccf56e5861b4cba8562031d12fcb535"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.958211 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:44.958331 31263 ts_manager.cc:194] Registered new tserver with Master: 7ab40f51249245bcacf1639206e8c30b (127.30.128.65:46701)
I20251026 03:07:44.959098 31263 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44548
I20251026 03:07:44.973489 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:44.973742 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:44.973897 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:07:44.974133 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:07:44.974166 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.974187 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:07:44.974215 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.977494 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:40621
I20251026 03:07:44.977559 31468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:40621 every 8 connection(s)
I20251026 03:07:44.977885 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:44.978983 31473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.979103 31475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:44.979215 31472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:44.979554 31233 server_base.cc:1047] running on GCE node
I20251026 03:07:44.979727 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:44.979787 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:44.979833 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448064979833 us; error 0 us; skew 500 ppm
I20251026 03:07:44.980484 31233 webserver.cc:492] Webserver started at http://127.30.128.67:34651/ using document root <none> and password file <none>
I20251026 03:07:44.980594 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:44.980655 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:44.980736 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:44.981042 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "c8e6e1abf8224ec689defa1e1a4cd404"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.981494 31469 heartbeater.cc:344] Connected to a master server at 127.30.128.126:41369
I20251026 03:07:44.981559 31469 heartbeater.cc:461] Registering TS with master...
I20251026 03:07:44.981657 31469 heartbeater.cc:507] Master 127.30.128.126:41369 requested a full tablet report, sending...
I20251026 03:07:44.981870 31263 ts_manager.cc:194] Registered new tserver with Master: 9ccf56e5861b4cba8562031d12fcb535 (127.30.128.66:40621)
I20251026 03:07:44.982254 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:07:44.982277 31263 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44556
I20251026 03:07:44.982831 31480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.982951 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:07:44.982997 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "c8e6e1abf8224ec689defa1e1a4cd404"
format_stamp: "Formatted at 2025-10-26 03:07:44 on dist-test-slave-51kf"
I20251026 03:07:44.983042 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestReplicateParticipantOps.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:44.997200 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:44.997452 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:44.997588 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:07:44.997747 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:07:44.997789 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:44.997817 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:07:44.997839 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:45.000912 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:34495
I20251026 03:07:45.001057 31543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:34495 every 8 connection(s)
I20251026 03:07:45.004356 31544 heartbeater.cc:344] Connected to a master server at 127.30.128.126:41369
I20251026 03:07:45.004420 31544 heartbeater.cc:461] Registering TS with master...
I20251026 03:07:45.004513 31544 heartbeater.cc:507] Master 127.30.128.126:41369 requested a full tablet report, sending...
I20251026 03:07:45.004707 31263 ts_manager.cc:194] Registered new tserver with Master: c8e6e1abf8224ec689defa1e1a4cd404 (127.30.128.67:34495)
I20251026 03:07:45.005035 31263 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44566
I20251026 03:07:45.005470 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004408498s
I20251026 03:07:45.005896 31233 test_util.cc:276] Using random seed: 372537670
I20251026 03:07:45.019630 31263 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44570:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:07:45.019865 31263 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:07:45.025532 31508 tablet_service.cc:1505] Processing CreateTablet for tablet a1232ee7f95f49869745244d5c636c5e (DEFAULT_TABLE table=test-workload [id=8cf1fabc039448499146746e37441089]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:07:45.025714 31508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1232ee7f95f49869745244d5c636c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:45.027092 31559 tablet_bootstrap.cc:492] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: Bootstrap starting.
I20251026 03:07:45.027161 31433 tablet_service.cc:1505] Processing CreateTablet for tablet a1232ee7f95f49869745244d5c636c5e (DEFAULT_TABLE table=test-workload [id=8cf1fabc039448499146746e37441089]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:07:45.027271 31433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1232ee7f95f49869745244d5c636c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:45.027525 31559 tablet_bootstrap.cc:654] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:45.028163 31559 tablet_bootstrap.cc:492] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: No bootstrap required, opened a new log
I20251026 03:07:45.028194 31559 ts_tablet_manager.cc:1403] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:45.028318 31559 raft_consensus.cc:359] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.028362 31559 raft_consensus.cc:385] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:45.028373 31559 raft_consensus.cc:740] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e6e1abf8224ec689defa1e1a4cd404, State: Initialized, Role: FOLLOWER
I20251026 03:07:45.028409 31559 consensus_queue.cc:260] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.028553 31559 ts_tablet_manager.cc:1434] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:45.028657 31544 heartbeater.cc:499] Master 127.30.128.126:41369 was elected leader, sending a full tablet report...
I20251026 03:07:45.029524 31357 tablet_service.cc:1505] Processing CreateTablet for tablet a1232ee7f95f49869745244d5c636c5e (DEFAULT_TABLE table=test-workload [id=8cf1fabc039448499146746e37441089]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:07:45.029659 31357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1232ee7f95f49869745244d5c636c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:45.029968 31561 tablet_bootstrap.cc:492] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535: Bootstrap starting.
I20251026 03:07:45.030436 31561 tablet_bootstrap.cc:654] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:45.030936 31564 tablet_bootstrap.cc:492] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: Bootstrap starting.
I20251026 03:07:45.031353 31561 tablet_bootstrap.cc:492] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535: No bootstrap required, opened a new log
I20251026 03:07:45.031440 31561 ts_tablet_manager.cc:1403] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:07:45.031563 31564 tablet_bootstrap.cc:654] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:45.031644 31561 raft_consensus.cc:359] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.031716 31561 raft_consensus.cc:385] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:45.031767 31561 raft_consensus.cc:740] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ccf56e5861b4cba8562031d12fcb535, State: Initialized, Role: FOLLOWER
I20251026 03:07:45.031839 31561 consensus_queue.cc:260] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.031992 31561 ts_tablet_manager.cc:1434] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:45.032047 31469 heartbeater.cc:499] Master 127.30.128.126:41369 was elected leader, sending a full tablet report...
I20251026 03:07:45.032317 31564 tablet_bootstrap.cc:492] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: No bootstrap required, opened a new log
I20251026 03:07:45.032392 31564 ts_tablet_manager.cc:1403] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:45.032529 31564 raft_consensus.cc:359] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.032626 31564 raft_consensus.cc:385] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:45.032666 31564 raft_consensus.cc:740] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab40f51249245bcacf1639206e8c30b, State: Initialized, Role: FOLLOWER
I20251026 03:07:45.032748 31564 consensus_queue.cc:260] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.032933 31393 heartbeater.cc:499] Master 127.30.128.126:41369 was elected leader, sending a full tablet report...
I20251026 03:07:45.032920 31564 ts_tablet_manager.cc:1434] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:45.169910 31562 raft_consensus.cc:493] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:07:45.170015 31562 raft_consensus.cc:515] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.170241 31562 leader_election.cc:290] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ab40f51249245bcacf1639206e8c30b (127.30.128.65:46701), 9ccf56e5861b4cba8562031d12fcb535 (127.30.128.66:40621)
I20251026 03:07:45.172667 31367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e" candidate_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab40f51249245bcacf1639206e8c30b" is_pre_election: true
I20251026 03:07:45.172724 31443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e" candidate_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf56e5861b4cba8562031d12fcb535" is_pre_election: true
I20251026 03:07:45.172778 31367 raft_consensus.cc:2468] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e6e1abf8224ec689defa1e1a4cd404 in term 0.
I20251026 03:07:45.172791 31443 raft_consensus.cc:2468] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8e6e1abf8224ec689defa1e1a4cd404 in term 0.
I20251026 03:07:45.172958 31482 leader_election.cc:304] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ab40f51249245bcacf1639206e8c30b, c8e6e1abf8224ec689defa1e1a4cd404; no voters:
I20251026 03:07:45.173087 31562 raft_consensus.cc:2804] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:07:45.173125 31562 raft_consensus.cc:493] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:07:45.173141 31562 raft_consensus.cc:3060] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:45.173784 31562 raft_consensus.cc:515] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.173902 31562 leader_election.cc:290] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [CANDIDATE]: Term 1 election: Requested vote from peers 7ab40f51249245bcacf1639206e8c30b (127.30.128.65:46701), 9ccf56e5861b4cba8562031d12fcb535 (127.30.128.66:40621)
I20251026 03:07:45.174015 31443 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e" candidate_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ccf56e5861b4cba8562031d12fcb535"
I20251026 03:07:45.174015 31367 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e" candidate_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab40f51249245bcacf1639206e8c30b"
I20251026 03:07:45.174108 31443 raft_consensus.cc:3060] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:45.174119 31367 raft_consensus.cc:3060] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:45.174809 31367 raft_consensus.cc:2468] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e6e1abf8224ec689defa1e1a4cd404 in term 1.
I20251026 03:07:45.174826 31443 raft_consensus.cc:2468] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8e6e1abf8224ec689defa1e1a4cd404 in term 1.
I20251026 03:07:45.174999 31482 leader_election.cc:304] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ab40f51249245bcacf1639206e8c30b, c8e6e1abf8224ec689defa1e1a4cd404; no voters:
I20251026 03:07:45.175098 31562 raft_consensus.cc:2804] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:07:45.175215 31562 raft_consensus.cc:697] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 LEADER]: Becoming Leader. State: Replica: c8e6e1abf8224ec689defa1e1a4cd404, State: Running, Role: LEADER
I20251026 03:07:45.175268 31562 consensus_queue.cc:237] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:45.175872 31263 catalog_manager.cc:5649] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8e6e1abf8224ec689defa1e1a4cd404 (127.30.128.67). New cstate: current_term: 1 leader_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } health_report { overall_health: UNKNOWN } } }
W20251026 03:07:45.201990 31394 tablet.cc:2378] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:07:45.209545 31443 raft_consensus.cc:1275] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 1 FOLLOWER]: Refusing update from remote peer c8e6e1abf8224ec689defa1e1a4cd404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:07:45.209549 31367 raft_consensus.cc:1275] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 1 FOLLOWER]: Refusing update from remote peer c8e6e1abf8224ec689defa1e1a4cd404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:07:45.209888 31569 consensus_queue.cc:1048] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:07:45.209992 31562 consensus_queue.cc:1048] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:07:45.211826 31577 mvcc.cc:204] Tried to move back new op lower bound from 7214891275096670208 to 7214891274958372864. Current Snapshot: MvccSnapshot[applied={T|T < 7214891275096670208}]
I20251026 03:07:55.709142 31569 raft_consensus.cc:993] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: : Instructing follower 7ab40f51249245bcacf1639206e8c30b to start an election
I20251026 03:07:55.709225 31569 raft_consensus.cc:1081] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 LEADER]: Signalling peer 7ab40f51249245bcacf1639206e8c30b to start an election
I20251026 03:07:55.709419 31366 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e"
dest_uuid: "7ab40f51249245bcacf1639206e8c30b"
from {username='slave'} at 127.0.0.1:49606
I20251026 03:07:55.709510 31366 raft_consensus.cc:493] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:07:55.709535 31366 raft_consensus.cc:3060] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:07:55.710206 31366 raft_consensus.cc:515] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:55.710363 31366 leader_election.cc:290] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [CANDIDATE]: Term 2 election: Requested vote from peers c8e6e1abf8224ec689defa1e1a4cd404 (127.30.128.67:34495), 9ccf56e5861b4cba8562031d12fcb535 (127.30.128.66:40621)
I20251026 03:07:55.712903 31517 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e" candidate_uuid: "7ab40f51249245bcacf1639206e8c30b" candidate_term: 2 candidate_status { last_received { term: 1 index: 14669 } } ignore_live_leader: true dest_uuid: "c8e6e1abf8224ec689defa1e1a4cd404"
I20251026 03:07:55.712963 31517 raft_consensus.cc:3055] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:07:55.712986 31517 raft_consensus.cc:740] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8e6e1abf8224ec689defa1e1a4cd404, State: Running, Role: LEADER
I20251026 03:07:55.713037 31442 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a1232ee7f95f49869745244d5c636c5e" candidate_uuid: "7ab40f51249245bcacf1639206e8c30b" candidate_term: 2 candidate_status { last_received { term: 1 index: 14669 } } ignore_live_leader: true dest_uuid: "9ccf56e5861b4cba8562031d12fcb535"
I20251026 03:07:55.713050 31517 consensus_queue.cc:260] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14669, Committed index: 14669, Last appended: 1.14669, Last appended by leader: 14669, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:55.713130 31442 raft_consensus.cc:3060] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:07:55.713188 31517 raft_consensus.cc:3060] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:07:55.713706 31442 raft_consensus.cc:2468] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ab40f51249245bcacf1639206e8c30b in term 2.
I20251026 03:07:55.713734 31517 raft_consensus.cc:2468] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ab40f51249245bcacf1639206e8c30b in term 2.
I20251026 03:07:55.713876 31333 leader_election.cc:304] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ab40f51249245bcacf1639206e8c30b, 9ccf56e5861b4cba8562031d12fcb535; no voters:
I20251026 03:07:55.714048 31624 raft_consensus.cc:2804] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:07:55.714161 31624 raft_consensus.cc:697] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 2 LEADER]: Becoming Leader. State: Replica: 7ab40f51249245bcacf1639206e8c30b, State: Running, Role: LEADER
I20251026 03:07:55.714217 31624 consensus_queue.cc:237] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14669, Committed index: 14669, Last appended: 1.14669, Last appended by leader: 14669, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } }
I20251026 03:07:55.714740 31263 catalog_manager.cc:5649] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b reported cstate change: term changed from 1 to 2, leader changed from c8e6e1abf8224ec689defa1e1a4cd404 (127.30.128.67) to 7ab40f51249245bcacf1639206e8c30b (127.30.128.65). New cstate: current_term: 2 leader_uuid: "7ab40f51249245bcacf1639206e8c30b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ab40f51249245bcacf1639206e8c30b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 } health_report { overall_health: UNKNOWN } } }
I20251026 03:07:55.714937 31442 raft_consensus.cc:1275] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 2 FOLLOWER]: Refusing update from remote peer 7ab40f51249245bcacf1639206e8c30b: Log matching property violated. Preceding OpId in replica: term: 1 index: 14669. Preceding OpId from leader: term: 2 index: 14671. (index mismatch)
I20251026 03:07:55.715001 31517 raft_consensus.cc:1275] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 2 FOLLOWER]: Refusing update from remote peer 7ab40f51249245bcacf1639206e8c30b: Log matching property violated. Preceding OpId in replica: term: 1 index: 14669. Preceding OpId from leader: term: 2 index: 14671. (index mismatch)
I20251026 03:07:55.715116 31624 consensus_queue.cc:1048] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ccf56e5861b4cba8562031d12fcb535" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14670, Last known committed idx: 14669, Time since last communication: 0.000s
I20251026 03:07:55.715262 31625 consensus_queue.cc:1048] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8e6e1abf8224ec689defa1e1a4cd404" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14670, Last known committed idx: 14669, Time since last communication: 0.000s
I20251026 03:07:55.720889 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:07:55.723392 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:07:55.723497 31233 tablet_replica.cc:333] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b: stopping tablet replica
I20251026 03:07:55.723569 31233 raft_consensus.cc:2243] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:07:55.723627 31233 raft_consensus.cc:2272] T a1232ee7f95f49869745244d5c636c5e P 7ab40f51249245bcacf1639206e8c30b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:07:55.736951 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:07:55.737797 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:07:55.740427 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:07:55.740550 31233 tablet_replica.cc:333] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535: stopping tablet replica
I20251026 03:07:55.740595 31233 raft_consensus.cc:2243] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:07:55.740636 31233 raft_consensus.cc:2272] T a1232ee7f95f49869745244d5c636c5e P 9ccf56e5861b4cba8562031d12fcb535 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:07:55.759110 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:07:55.759752 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:07:55.762224 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:07:55.762333 31233 tablet_replica.cc:333] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404: stopping tablet replica
I20251026 03:07:55.762389 31233 raft_consensus.cc:2243] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:07:55.762434 31233 raft_consensus.cc:2272] T a1232ee7f95f49869745244d5c636c5e P c8e6e1abf8224ec689defa1e1a4cd404 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:07:55.783831 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:07:55.784466 31233 master.cc:561] Master@127.30.128.126:41369 shutting down...
I20251026 03:07:55.786535 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:07:55.786594 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:07:55.786628 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P f72a49c0207c408ca32478cebd4763f7: stopping tablet replica
I20251026 03:07:55.798286 31233 master.cc:583] Master@127.30.128.126:41369 shutdown complete.
[ OK ] TxnParticipantITest.TestReplicateParticipantOps (10953 ms)
[ RUN ] TxnParticipantITest.TestWaitOnFinalizeCommit
I20251026 03:07:55.816322 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:41289
I20251026 03:07:55.816468 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:55.817886 31663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:55.817929 31233 server_base.cc:1047] running on GCE node
W20251026 03:07:55.818011 31662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:55.817936 31665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:55.818286 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:55.818323 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:55.818336 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448075818337 us; error 0 us; skew 500 ppm
I20251026 03:07:55.818831 31233 webserver.cc:492] Webserver started at http://127.30.128.126:46179/ using document root <none> and password file <none>
I20251026 03:07:55.818892 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:55.818946 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:55.818993 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:55.819214 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "ee3109be5dd74c7697c8be28993b9910"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.819968 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:55.820326 31670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.820464 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.820510 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "ee3109be5dd74c7697c8be28993b9910"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.820551 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:55.828204 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:55.828323 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:55.830927 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:41289
I20251026 03:07:55.832510 31722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:41289 every 8 connection(s)
I20251026 03:07:55.832648 31723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:55.833501 31723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910: Bootstrap starting.
I20251026 03:07:55.833724 31723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:55.834142 31723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910: No bootstrap required, opened a new log
I20251026 03:07:55.834232 31723 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3109be5dd74c7697c8be28993b9910" member_type: VOTER }
I20251026 03:07:55.834270 31723 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:55.834290 31723 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3109be5dd74c7697c8be28993b9910, State: Initialized, Role: FOLLOWER
I20251026 03:07:55.834326 31723 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3109be5dd74c7697c8be28993b9910" member_type: VOTER }
I20251026 03:07:55.834363 31723 raft_consensus.cc:399] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:07:55.834389 31723 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:07:55.834407 31723 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:55.834800 31723 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3109be5dd74c7697c8be28993b9910" member_type: VOTER }
I20251026 03:07:55.834846 31723 leader_election.cc:304] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3109be5dd74c7697c8be28993b9910; no voters:
I20251026 03:07:55.834955 31723 leader_election.cc:290] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:07:55.835004 31726 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:07:55.835093 31723 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:07:55.835124 31726 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 1 LEADER]: Becoming Leader. State: Replica: ee3109be5dd74c7697c8be28993b9910, State: Running, Role: LEADER
I20251026 03:07:55.835177 31726 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3109be5dd74c7697c8be28993b9910" member_type: VOTER }
I20251026 03:07:55.835388 31727 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee3109be5dd74c7697c8be28993b9910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3109be5dd74c7697c8be28993b9910" member_type: VOTER } }
I20251026 03:07:55.835412 31728 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3109be5dd74c7697c8be28993b9910. Latest consensus state: current_term: 1 leader_uuid: "ee3109be5dd74c7697c8be28993b9910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3109be5dd74c7697c8be28993b9910" member_type: VOTER } }
I20251026 03:07:55.835459 31727 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [sys.catalog]: This master's current role is: LEADER
I20251026 03:07:55.835470 31728 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [sys.catalog]: This master's current role is: LEADER
I20251026 03:07:55.835793 31734 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:07:55.835994 31734 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:07:55.836290 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:07:55.836503 31734 catalog_manager.cc:1357] Generated new cluster ID: ed061d24ac564455b128b7ca89343c27
I20251026 03:07:55.836542 31734 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:07:55.843299 31734 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:07:55.843667 31734 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:07:55.850006 31734 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910: Generated new TSK 0
I20251026 03:07:55.850078 31734 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:07:55.851889 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:55.852923 31744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:55.852994 31745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:55.853039 31747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:55.853124 31233 server_base.cc:1047] running on GCE node
I20251026 03:07:55.853255 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:55.853286 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:55.853303 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448075853303 us; error 0 us; skew 500 ppm
I20251026 03:07:55.853819 31233 webserver.cc:492] Webserver started at http://127.30.128.65:37343/ using document root <none> and password file <none>
I20251026 03:07:55.853894 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:55.853943 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:55.853987 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:55.854200 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "bc1b0a8d0fb748b9a563026419f825d4"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.855023 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:07:55.855443 31752 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.855563 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.855602 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "bc1b0a8d0fb748b9a563026419f825d4"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.855645 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:55.877805 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:55.877976 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:55.878126 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:07:55.878320 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:07:55.878345 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.878377 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:07:55.878403 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.880910 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:42531
I20251026 03:07:55.881454 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:07:55.881903 31815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:42531 every 8 connection(s)
W20251026 03:07:55.882969 31819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:55.882989 31823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:55.882968 31818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:55.883350 31233 server_base.cc:1047] running on GCE node
I20251026 03:07:55.883432 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:55.883463 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:55.883491 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448075883491 us; error 0 us; skew 500 ppm
I20251026 03:07:55.883996 31233 webserver.cc:492] Webserver started at http://127.30.128.66:34269/ using document root <none> and password file <none>
I20251026 03:07:55.884079 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:55.884120 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:55.884162 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:55.884485 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "2b360f9878d54820a09544bc5b87ccb6"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.885259 31816 heartbeater.cc:344] Connected to a master server at 127.30.128.126:41289
I20251026 03:07:55.885315 31816 heartbeater.cc:461] Registering TS with master...
I20251026 03:07:55.885413 31816 heartbeater.cc:507] Master 127.30.128.126:41289 requested a full tablet report, sending...
I20251026 03:07:55.885489 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:55.885637 31687 ts_manager.cc:194] Registered new tserver with Master: bc1b0a8d0fb748b9a563026419f825d4 (127.30.128.65:42531)
I20251026 03:07:55.885941 31828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.886041 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.886081 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "2b360f9878d54820a09544bc5b87ccb6"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.886121 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:55.886269 31687 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41070
I20251026 03:07:55.902372 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:55.902536 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:55.902637 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:07:55.902782 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:07:55.902806 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.902824 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:07:55.902848 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.905422 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:40471
I20251026 03:07:55.905490 31891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:40471 every 8 connection(s)
I20251026 03:07:55.905740 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:07:55.906816 31895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:07:55.906852 31896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:55.907032 31233 server_base.cc:1047] running on GCE node
W20251026 03:07:55.907083 31898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:07:55.907241 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:07:55.907282 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:07:55.907299 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448075907300 us; error 0 us; skew 500 ppm
I20251026 03:07:55.907868 31233 webserver.cc:492] Webserver started at http://127.30.128.67:38259/ using document root <none> and password file <none>
I20251026 03:07:55.907955 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:07:55.908003 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:07:55.908049 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:07:55.908308 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "aa67015519734937a0c488a8775a2352"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.908447 31892 heartbeater.cc:344] Connected to a master server at 127.30.128.126:41289
I20251026 03:07:55.908517 31892 heartbeater.cc:461] Registering TS with master...
I20251026 03:07:55.908622 31892 heartbeater.cc:507] Master 127.30.128.126:41289 requested a full tablet report, sending...
I20251026 03:07:55.908860 31687 ts_manager.cc:194] Registered new tserver with Master: 2b360f9878d54820a09544bc5b87ccb6 (127.30.128.66:40471)
I20251026 03:07:55.909261 31687 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41074
I20251026 03:07:55.909292 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:55.909811 31903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.909914 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.909955 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "aa67015519734937a0c488a8775a2352"
format_stamp: "Formatted at 2025-10-26 03:07:55 on dist-test-slave-51kf"
I20251026 03:07:55.909987 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnFinalizeCommit.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:07:55.913836 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:07:55.913995 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:07:55.914091 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:07:55.914201 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:07:55.914219 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.914240 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:07:55.914255 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.917207 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:34351
I20251026 03:07:55.917250 31966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:34351 every 8 connection(s)
I20251026 03:07:55.919788 31967 heartbeater.cc:344] Connected to a master server at 127.30.128.126:41289
I20251026 03:07:55.919835 31967 heartbeater.cc:461] Registering TS with master...
I20251026 03:07:55.919920 31967 heartbeater.cc:507] Master 127.30.128.126:41289 requested a full tablet report, sending...
I20251026 03:07:55.920107 31687 ts_manager.cc:194] Registered new tserver with Master: aa67015519734937a0c488a8775a2352 (127.30.128.67:34351)
I20251026 03:07:55.920420 31687 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41086
I20251026 03:07:55.920655 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003301937s
I20251026 03:07:55.920979 31233 test_util.cc:276] Using random seed: 383452755
I20251026 03:07:55.925098 31687 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41094:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:07:55.925220 31687 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:07:55.929265 31780 tablet_service.cc:1505] Processing CreateTablet for tablet 02910e22f725494189c9212339ecb0e2 (DEFAULT_TABLE table=test-workload [id=3033383bb187477daf0e93d6f0552ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:07:55.929365 31780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02910e22f725494189c9212339ecb0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:55.929612 31931 tablet_service.cc:1505] Processing CreateTablet for tablet 02910e22f725494189c9212339ecb0e2 (DEFAULT_TABLE table=test-workload [id=3033383bb187477daf0e93d6f0552ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:07:55.929690 31856 tablet_service.cc:1505] Processing CreateTablet for tablet 02910e22f725494189c9212339ecb0e2 (DEFAULT_TABLE table=test-workload [id=3033383bb187477daf0e93d6f0552ce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:07:55.929700 31931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02910e22f725494189c9212339ecb0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:55.929836 31856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02910e22f725494189c9212339ecb0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:07:55.930851 31982 tablet_bootstrap.cc:492] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4: Bootstrap starting.
I20251026 03:07:55.931079 31984 tablet_bootstrap.cc:492] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352: Bootstrap starting.
I20251026 03:07:55.931279 31983 tablet_bootstrap.cc:492] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6: Bootstrap starting.
I20251026 03:07:55.931316 31982 tablet_bootstrap.cc:654] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:55.931545 31984 tablet_bootstrap.cc:654] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:55.931740 31983 tablet_bootstrap.cc:654] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6: Neither blocks nor log segments found. Creating new log.
I20251026 03:07:55.931893 31982 tablet_bootstrap.cc:492] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4: No bootstrap required, opened a new log
I20251026 03:07:55.931926 31982 ts_tablet_manager.cc:1403] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:55.932070 31982 raft_consensus.cc:359] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:55.932111 31984 tablet_bootstrap.cc:492] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352: No bootstrap required, opened a new log
I20251026 03:07:55.932126 31982 raft_consensus.cc:385] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:55.932147 31982 raft_consensus.cc:740] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc1b0a8d0fb748b9a563026419f825d4, State: Initialized, Role: FOLLOWER
I20251026 03:07:55.932224 31982 consensus_queue.cc:260] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:55.932282 31983 tablet_bootstrap.cc:492] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6: No bootstrap required, opened a new log
I20251026 03:07:55.932329 31983 ts_tablet_manager.cc:1403] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:55.932381 31982 ts_tablet_manager.cc:1434] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:07:55.932408 31816 heartbeater.cc:499] Master 127.30.128.126:41289 was elected leader, sending a full tablet report...
I20251026 03:07:55.932140 31984 ts_tablet_manager.cc:1403] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:07:55.932473 31983 raft_consensus.cc:359] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:55.932525 31983 raft_consensus.cc:385] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:55.932544 31983 raft_consensus.cc:740] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b360f9878d54820a09544bc5b87ccb6, State: Initialized, Role: FOLLOWER
I20251026 03:07:55.932575 31984 raft_consensus.cc:359] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:55.932617 31983 consensus_queue.cc:260] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:55.932638 31984 raft_consensus.cc:385] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:07:55.932689 31984 raft_consensus.cc:740] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67015519734937a0c488a8775a2352, State: Initialized, Role: FOLLOWER
I20251026 03:07:55.932746 31984 consensus_queue.cc:260] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:55.932785 31983 ts_tablet_manager.cc:1434] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:07:55.932854 31892 heartbeater.cc:499] Master 127.30.128.126:41289 was elected leader, sending a full tablet report...
I20251026 03:07:55.932893 31984 ts_tablet_manager.cc:1434] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:07:55.933161 31967 heartbeater.cc:499] Master 127.30.128.126:41289 was elected leader, sending a full tablet report...
I20251026 03:07:56.122648 31988 raft_consensus.cc:493] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:07:56.122741 31988 raft_consensus.cc:515] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:56.122931 31988 leader_election.cc:290] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa67015519734937a0c488a8775a2352 (127.30.128.67:34351), 2b360f9878d54820a09544bc5b87ccb6 (127.30.128.66:40471)
I20251026 03:07:56.125617 31941 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02910e22f725494189c9212339ecb0e2" candidate_uuid: "bc1b0a8d0fb748b9a563026419f825d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67015519734937a0c488a8775a2352" is_pre_election: true
I20251026 03:07:56.125692 31941 raft_consensus.cc:2468] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc1b0a8d0fb748b9a563026419f825d4 in term 0.
I20251026 03:07:56.125694 31866 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02910e22f725494189c9212339ecb0e2" candidate_uuid: "bc1b0a8d0fb748b9a563026419f825d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b360f9878d54820a09544bc5b87ccb6" is_pre_election: true
I20251026 03:07:56.125826 31866 raft_consensus.cc:2468] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc1b0a8d0fb748b9a563026419f825d4 in term 0.
I20251026 03:07:56.125845 31755 leader_election.cc:304] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa67015519734937a0c488a8775a2352, bc1b0a8d0fb748b9a563026419f825d4; no voters:
I20251026 03:07:56.125916 31988 raft_consensus.cc:2804] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:07:56.125953 31988 raft_consensus.cc:493] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:07:56.125970 31988 raft_consensus.cc:3060] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:56.127714 31988 raft_consensus.cc:515] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:56.127812 31988 leader_election.cc:290] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [CANDIDATE]: Term 1 election: Requested vote from peers aa67015519734937a0c488a8775a2352 (127.30.128.67:34351), 2b360f9878d54820a09544bc5b87ccb6 (127.30.128.66:40471)
I20251026 03:07:56.127971 31866 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02910e22f725494189c9212339ecb0e2" candidate_uuid: "bc1b0a8d0fb748b9a563026419f825d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b360f9878d54820a09544bc5b87ccb6"
I20251026 03:07:56.127971 31941 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02910e22f725494189c9212339ecb0e2" candidate_uuid: "bc1b0a8d0fb748b9a563026419f825d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67015519734937a0c488a8775a2352"
I20251026 03:07:56.128055 31866 raft_consensus.cc:3060] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:56.128059 31941 raft_consensus.cc:3060] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:07:56.128688 31941 raft_consensus.cc:2468] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc1b0a8d0fb748b9a563026419f825d4 in term 1.
I20251026 03:07:56.128688 31866 raft_consensus.cc:2468] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc1b0a8d0fb748b9a563026419f825d4 in term 1.
I20251026 03:07:56.128881 31756 leader_election.cc:304] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b360f9878d54820a09544bc5b87ccb6, bc1b0a8d0fb748b9a563026419f825d4; no voters:
I20251026 03:07:56.128978 31988 raft_consensus.cc:2804] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:07:56.129119 31988 raft_consensus.cc:697] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 1 LEADER]: Becoming Leader. State: Replica: bc1b0a8d0fb748b9a563026419f825d4, State: Running, Role: LEADER
I20251026 03:07:56.129215 31988 consensus_queue.cc:237] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } }
I20251026 03:07:56.129724 31685 catalog_manager.cc:5649] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc1b0a8d0fb748b9a563026419f825d4 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "bc1b0a8d0fb748b9a563026419f825d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc1b0a8d0fb748b9a563026419f825d4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 } health_report { overall_health: UNKNOWN } } }
I20251026 03:07:56.159555 31866 raft_consensus.cc:1275] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 1 FOLLOWER]: Refusing update from remote peer bc1b0a8d0fb748b9a563026419f825d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:07:56.159570 31941 raft_consensus.cc:1275] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 1 FOLLOWER]: Refusing update from remote peer bc1b0a8d0fb748b9a563026419f825d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:07:56.159763 31992 consensus_queue.cc:1048] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa67015519734937a0c488a8775a2352" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:07:56.159844 31988 consensus_queue.cc:1048] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b360f9878d54820a09544bc5b87ccb6" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:08.217931 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:08:08.220188 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:08.220326 31233 tablet_replica.cc:333] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4: stopping tablet replica
I20251026 03:08:08.220394 31233 raft_consensus.cc:2243] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:08.220455 31233 raft_consensus.cc:2272] T 02910e22f725494189c9212339ecb0e2 P bc1b0a8d0fb748b9a563026419f825d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:08.234270 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:08:08.235049 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:08:08.237534 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:08.237639 31233 tablet_replica.cc:333] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6: stopping tablet replica
I20251026 03:08:08.237699 31233 raft_consensus.cc:2243] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:08.237746 31233 raft_consensus.cc:2272] T 02910e22f725494189c9212339ecb0e2 P 2b360f9878d54820a09544bc5b87ccb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:08.256469 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:08:08.257288 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:08:08.259994 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:08.260155 31233 tablet_replica.cc:333] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352: stopping tablet replica
I20251026 03:08:08.260211 31233 raft_consensus.cc:2243] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:08.260262 31233 raft_consensus.cc:2272] T 02910e22f725494189c9212339ecb0e2 P aa67015519734937a0c488a8775a2352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:08.283098 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:08:08.283953 31233 master.cc:561] Master@127.30.128.126:41289 shutting down...
I20251026 03:08:08.286861 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:08.286923 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:08.286942 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P ee3109be5dd74c7697c8be28993b9910: stopping tablet replica
I20251026 03:08:08.298434 31233 master.cc:583] Master@127.30.128.126:41289 shutdown complete.
[ OK ] TxnParticipantITest.TestWaitOnFinalizeCommit (12502 ms)
[ RUN ] TxnParticipantITest.TestWaitOnAbortCommit
I20251026 03:08:08.318714 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:38471
I20251026 03:08:08.318879 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:08.320247 32071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.320315 32068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:08.320492 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:08.320611 32069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:08.320734 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:08.320770 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:08.320783 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448088320784 us; error 0 us; skew 500 ppm
I20251026 03:08:08.321344 31233 webserver.cc:492] Webserver started at http://127.30.128.126:42091/ using document root <none> and password file <none>
I20251026 03:08:08.321424 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:08.321475 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:08.321527 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:08.321810 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "d784da9a147d4e23b4f0dccf2719cdfc"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.322788 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:08.323216 32076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.323336 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.323377 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "d784da9a147d4e23b4f0dccf2719cdfc"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.323424 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:08.333761 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:08.333962 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:08.336323 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:38471
I20251026 03:08:08.337713 32129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:08.338954 32128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:38471 every 8 connection(s)
I20251026 03:08:08.339751 32129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc: Bootstrap starting.
I20251026 03:08:08.340054 32129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:08.340464 32129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc: No bootstrap required, opened a new log
I20251026 03:08:08.340579 32129 raft_consensus.cc:359] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" member_type: VOTER }
I20251026 03:08:08.340639 32129 raft_consensus.cc:385] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:08.340682 32129 raft_consensus.cc:740] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d784da9a147d4e23b4f0dccf2719cdfc, State: Initialized, Role: FOLLOWER
I20251026 03:08:08.340742 32129 consensus_queue.cc:260] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" member_type: VOTER }
I20251026 03:08:08.340780 32129 raft_consensus.cc:399] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:08:08.340806 32129 raft_consensus.cc:493] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:08:08.340834 32129 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:08.355640 32129 raft_consensus.cc:515] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" member_type: VOTER }
I20251026 03:08:08.355743 32129 leader_election.cc:304] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d784da9a147d4e23b4f0dccf2719cdfc; no voters:
I20251026 03:08:08.355849 32129 leader_election.cc:290] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:08:08.355948 32132 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:08.356011 32129 sys_catalog.cc:565] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:08:08.356144 32132 raft_consensus.cc:697] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 1 LEADER]: Becoming Leader. State: Replica: d784da9a147d4e23b4f0dccf2719cdfc, State: Running, Role: LEADER
I20251026 03:08:08.356209 32132 consensus_queue.cc:237] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" member_type: VOTER }
I20251026 03:08:08.356607 32133 sys_catalog.cc:455] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" member_type: VOTER } }
I20251026 03:08:08.356683 32133 sys_catalog.cc:458] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:08.356801 32134 sys_catalog.cc:455] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d784da9a147d4e23b4f0dccf2719cdfc. Latest consensus state: current_term: 1 leader_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d784da9a147d4e23b4f0dccf2719cdfc" member_type: VOTER } }
I20251026 03:08:08.356849 32134 sys_catalog.cc:458] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:08.356981 32141 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:08:08.357131 32141 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:08:08.357363 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:08:08.357904 32141 catalog_manager.cc:1357] Generated new cluster ID: 501921215cd74569929a00a951e9fd4a
I20251026 03:08:08.357955 32141 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:08:08.398838 32141 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:08:08.399287 32141 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:08:08.403334 32141 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc: Generated new TSK 0
I20251026 03:08:08.403431 32141 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:08:08.421756 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:08.423170 32153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.423264 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.423283 32150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:08.423549 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:08.423631 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:08.423662 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:08.423683 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448088423682 us; error 0 us; skew 500 ppm
I20251026 03:08:08.424320 31233 webserver.cc:492] Webserver started at http://127.30.128.65:41671/ using document root <none> and password file <none>
I20251026 03:08:08.424403 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:08.424453 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:08.424507 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:08.424787 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "07c5c993e37b4ff19d0c7385a5532f23"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.425591 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.426079 32158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.426203 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.426244 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "07c5c993e37b4ff19d0c7385a5532f23"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.426294 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:08.433916 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:08.434091 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:08.434226 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:08.434409 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:08.434437 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.434470 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:08.434496 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.437137 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:35993
I20251026 03:08:08.437561 32221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:35993 every 8 connection(s)
I20251026 03:08:08.437736 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:08.438728 32229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.438815 32227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.438742 32226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:08.439492 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:08.439610 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:08.439649 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:08.439679 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448088439678 us; error 0 us; skew 500 ppm
I20251026 03:08:08.440310 31233 webserver.cc:492] Webserver started at http://127.30.128.66:43469/ using document root <none> and password file <none>
I20251026 03:08:08.440392 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:08.440435 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:08.440474 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:08.440766 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.441427 32222 heartbeater.cc:344] Connected to a master server at 127.30.128.126:38471
I20251026 03:08:08.441488 32222 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:08.441612 32222 heartbeater.cc:507] Master 127.30.128.126:38471 requested a full tablet report, sending...
I20251026 03:08:08.441869 32093 ts_manager.cc:194] Registered new tserver with Master: 07c5c993e37b4ff19d0c7385a5532f23 (127.30.128.65:35993)
I20251026 03:08:08.441903 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.442444 32234 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.442480 32093 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35132
I20251026 03:08:08.442601 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 03:08:08.442646 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.442685 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:08.465060 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:08.465266 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:08.465428 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:08.465615 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:08.465641 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.465675 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:08.465706 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.468745 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:34025
I20251026 03:08:08.468789 32297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:34025 every 8 connection(s)
I20251026 03:08:08.469116 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:08.470209 32302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.470225 32301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:08.470394 32304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:08.470566 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:08.470678 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:08.470716 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:08.470729 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448088470730 us; error 0 us; skew 500 ppm
I20251026 03:08:08.471364 31233 webserver.cc:492] Webserver started at http://127.30.128.67:43373/ using document root <none> and password file <none>
I20251026 03:08:08.471458 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:08.471508 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:08.471555 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:08.471851 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "53a76c55d0cf4ef0b96ff3f3cc378427"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.472721 32298 heartbeater.cc:344] Connected to a master server at 127.30.128.126:38471
I20251026 03:08:08.472784 32298 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:08.472879 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.472888 32298 heartbeater.cc:507] Master 127.30.128.126:38471 requested a full tablet report, sending...
I20251026 03:08:08.473151 32093 ts_manager.cc:194] Registered new tserver with Master: ac61aa4ab8fe4e098bf01e7dc67da7af (127.30.128.66:34025)
I20251026 03:08:08.473405 32309 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.473536 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.473587 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "53a76c55d0cf4ef0b96ff3f3cc378427"
format_stamp: "Formatted at 2025-10-26 03:08:08 on dist-test-slave-51kf"
I20251026 03:08:08.473589 32093 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35142
I20251026 03:08:08.473650 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestWaitOnAbortCommit.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:08.484174 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:08.484364 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:08.484503 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:08.484681 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:08.484850 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.484915 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:08.484944 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.488179 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:46017
I20251026 03:08:08.488246 32372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:46017 every 8 connection(s)
I20251026 03:08:08.490983 32373 heartbeater.cc:344] Connected to a master server at 127.30.128.126:38471
I20251026 03:08:08.491034 32373 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:08.491125 32373 heartbeater.cc:507] Master 127.30.128.126:38471 requested a full tablet report, sending...
I20251026 03:08:08.491304 32093 ts_manager.cc:194] Registered new tserver with Master: 53a76c55d0cf4ef0b96ff3f3cc378427 (127.30.128.67:46017)
I20251026 03:08:08.491606 32093 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35150
I20251026 03:08:08.491621 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003273956s
I20251026 03:08:08.491952 31233 test_util.cc:276] Using random seed: 396023728
I20251026 03:08:08.495922 32093 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35160:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:08:08.496065 32093 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:08:08.500507 32262 tablet_service.cc:1505] Processing CreateTablet for tablet 8b9869a7c7144ee5859802482c30f375 (DEFAULT_TABLE table=test-workload [id=0d4c7371897445659818e7ceeba819c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:08.500645 32262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9869a7c7144ee5859802482c30f375. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:08.500934 32186 tablet_service.cc:1505] Processing CreateTablet for tablet 8b9869a7c7144ee5859802482c30f375 (DEFAULT_TABLE table=test-workload [id=0d4c7371897445659818e7ceeba819c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:08.501039 32186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9869a7c7144ee5859802482c30f375. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:08.502269 32337 tablet_service.cc:1505] Processing CreateTablet for tablet 8b9869a7c7144ee5859802482c30f375 (DEFAULT_TABLE table=test-workload [id=0d4c7371897445659818e7ceeba819c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:08.502346 32388 tablet_bootstrap.cc:492] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af: Bootstrap starting.
I20251026 03:08:08.502382 32337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b9869a7c7144ee5859802482c30f375. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:08.502504 32389 tablet_bootstrap.cc:492] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23: Bootstrap starting.
I20251026 03:08:08.502774 32388 tablet_bootstrap.cc:654] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:08.502980 32389 tablet_bootstrap.cc:654] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:08.503321 32388 tablet_bootstrap.cc:492] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af: No bootstrap required, opened a new log
I20251026 03:08:08.503389 32388 ts_tablet_manager.cc:1403] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.503571 32388 raft_consensus.cc:359] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.503631 32388 raft_consensus.cc:385] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:08.503644 32389 tablet_bootstrap.cc:492] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23: No bootstrap required, opened a new log
I20251026 03:08:08.503654 32388 raft_consensus.cc:740] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac61aa4ab8fe4e098bf01e7dc67da7af, State: Initialized, Role: FOLLOWER
I20251026 03:08:08.503729 32389 ts_tablet_manager.cc:1403] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.503782 32388 consensus_queue.cc:260] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.503888 32389 raft_consensus.cc:359] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.503988 32298 heartbeater.cc:499] Master 127.30.128.126:38471 was elected leader, sending a full tablet report...
I20251026 03:08:08.503958 32389 raft_consensus.cc:385] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:08.504040 32389 raft_consensus.cc:740] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c5c993e37b4ff19d0c7385a5532f23, State: Initialized, Role: FOLLOWER
I20251026 03:08:08.504089 32389 consensus_queue.cc:260] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.503975 32388 ts_tablet_manager.cc:1434] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.504238 32389 ts_tablet_manager.cc:1434] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:08:08.504256 32392 tablet_bootstrap.cc:492] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427: Bootstrap starting.
I20251026 03:08:08.504307 32222 heartbeater.cc:499] Master 127.30.128.126:38471 was elected leader, sending a full tablet report...
I20251026 03:08:08.504772 32392 tablet_bootstrap.cc:654] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:08.505342 32392 tablet_bootstrap.cc:492] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427: No bootstrap required, opened a new log
I20251026 03:08:08.505393 32392 ts_tablet_manager.cc:1403] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:08.505518 32392 raft_consensus.cc:359] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.505571 32392 raft_consensus.cc:385] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:08.505585 32392 raft_consensus.cc:740] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a76c55d0cf4ef0b96ff3f3cc378427, State: Initialized, Role: FOLLOWER
I20251026 03:08:08.505652 32392 consensus_queue.cc:260] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.505833 32392 ts_tablet_manager.cc:1434] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:08.505854 32373 heartbeater.cc:499] Master 127.30.128.126:38471 was elected leader, sending a full tablet report...
I20251026 03:08:08.557370 32394 raft_consensus.cc:493] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:08:08.557467 32394 raft_consensus.cc:515] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.557682 32394 leader_election.cc:290] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac61aa4ab8fe4e098bf01e7dc67da7af (127.30.128.66:34025), 53a76c55d0cf4ef0b96ff3f3cc378427 (127.30.128.67:46017)
I20251026 03:08:08.560061 32272 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b9869a7c7144ee5859802482c30f375" candidate_uuid: "07c5c993e37b4ff19d0c7385a5532f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" is_pre_election: true
I20251026 03:08:08.560123 32347 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b9869a7c7144ee5859802482c30f375" candidate_uuid: "07c5c993e37b4ff19d0c7385a5532f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" is_pre_election: true
I20251026 03:08:08.560163 32347 raft_consensus.cc:2468] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c5c993e37b4ff19d0c7385a5532f23 in term 0.
I20251026 03:08:08.560163 32272 raft_consensus.cc:2468] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c5c993e37b4ff19d0c7385a5532f23 in term 0.
I20251026 03:08:08.560326 32161 leader_election.cc:304] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c5c993e37b4ff19d0c7385a5532f23, ac61aa4ab8fe4e098bf01e7dc67da7af; no voters:
I20251026 03:08:08.560431 32394 raft_consensus.cc:2804] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:08:08.560480 32394 raft_consensus.cc:493] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:08:08.560513 32394 raft_consensus.cc:3060] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:08.652276 32394 raft_consensus.cc:515] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.652597 32394 leader_election.cc:290] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [CANDIDATE]: Term 1 election: Requested vote from peers ac61aa4ab8fe4e098bf01e7dc67da7af (127.30.128.66:34025), 53a76c55d0cf4ef0b96ff3f3cc378427 (127.30.128.67:46017)
I20251026 03:08:08.652696 32272 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b9869a7c7144ee5859802482c30f375" candidate_uuid: "07c5c993e37b4ff19d0c7385a5532f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af"
I20251026 03:08:08.652722 32347 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8b9869a7c7144ee5859802482c30f375" candidate_uuid: "07c5c993e37b4ff19d0c7385a5532f23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427"
I20251026 03:08:08.652812 32272 raft_consensus.cc:3060] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:08.652823 32347 raft_consensus.cc:3060] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:08.653504 32347 raft_consensus.cc:2468] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c5c993e37b4ff19d0c7385a5532f23 in term 1.
I20251026 03:08:08.653689 32159 leader_election.cc:304] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c5c993e37b4ff19d0c7385a5532f23, 53a76c55d0cf4ef0b96ff3f3cc378427; no voters:
I20251026 03:08:08.653797 32394 raft_consensus.cc:2804] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:08.653859 32272 raft_consensus.cc:2468] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c5c993e37b4ff19d0c7385a5532f23 in term 1.
I20251026 03:08:08.653939 32394 raft_consensus.cc:697] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 1 LEADER]: Becoming Leader. State: Replica: 07c5c993e37b4ff19d0c7385a5532f23, State: Running, Role: LEADER
I20251026 03:08:08.654008 32394 consensus_queue.cc:237] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } }
I20251026 03:08:08.654667 32093 catalog_manager.cc:5649] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c5c993e37b4ff19d0c7385a5532f23 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "07c5c993e37b4ff19d0c7385a5532f23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c5c993e37b4ff19d0c7385a5532f23" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 } health_report { overall_health: UNKNOWN } } }
I20251026 03:08:08.684684 32347 raft_consensus.cc:1275] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 1 FOLLOWER]: Refusing update from remote peer 07c5c993e37b4ff19d0c7385a5532f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:08.684702 32272 raft_consensus.cc:1275] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 1 FOLLOWER]: Refusing update from remote peer 07c5c993e37b4ff19d0c7385a5532f23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:08.684959 32394 consensus_queue.cc:1048] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac61aa4ab8fe4e098bf01e7dc67da7af" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:08.685048 32398 consensus_queue.cc:1048] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53a76c55d0cf4ef0b96ff3f3cc378427" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:19.722900 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:08:19.725756 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:19.725900 31233 tablet_replica.cc:333] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23: stopping tablet replica
I20251026 03:08:19.725955 31233 raft_consensus.cc:2243] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:19.726029 31233 raft_consensus.cc:2272] T 8b9869a7c7144ee5859802482c30f375 P 07c5c993e37b4ff19d0c7385a5532f23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:19.739316 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:08:19.740245 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:08:19.742832 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:19.742944 31233 tablet_replica.cc:333] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af: stopping tablet replica
I20251026 03:08:19.743005 31233 raft_consensus.cc:2243] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:19.743057 31233 raft_consensus.cc:2272] T 8b9869a7c7144ee5859802482c30f375 P ac61aa4ab8fe4e098bf01e7dc67da7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:19.761577 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:08:19.762501 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:08:19.765180 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:19.765300 31233 tablet_replica.cc:333] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427: stopping tablet replica
I20251026 03:08:19.765336 31233 raft_consensus.cc:2243] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:19.765370 31233 raft_consensus.cc:2272] T 8b9869a7c7144ee5859802482c30f375 P 53a76c55d0cf4ef0b96ff3f3cc378427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:19.787495 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:08:19.788327 31233 master.cc:561] Master@127.30.128.126:38471 shutting down...
I20251026 03:08:19.791270 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:19.791348 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:19.791368 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P d784da9a147d4e23b4f0dccf2719cdfc: stopping tablet replica
I20251026 03:08:19.803205 31233 master.cc:583] Master@127.30.128.126:38471 shutdown complete.
[ OK ] TxnParticipantITest.TestWaitOnAbortCommit (11504 ms)
[ RUN ] TxnParticipantITest.TestProxyBasicCalls
I20251026 03:08:19.822753 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:33859
I20251026 03:08:19.822925 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:19.824128 32454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.824297 32453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.824306 32456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:19.824446 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:19.824561 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:19.824601 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:19.824615 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448099824615 us; error 0 us; skew 500 ppm
I20251026 03:08:19.825145 31233 webserver.cc:492] Webserver started at http://127.30.128.126:44673/ using document root <none> and password file <none>
I20251026 03:08:19.825212 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:19.825245 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:19.825274 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:19.825493 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "1b20783b767d4e12896e1583e6cc0a32"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.826335 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:19.826723 32461 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.826890 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.826931 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "1b20783b767d4e12896e1583e6cc0a32"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.826982 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:19.839043 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:19.839226 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:19.841656 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:33859
I20251026 03:08:19.843618 32513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:33859 every 8 connection(s)
I20251026 03:08:19.843636 32514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:19.844600 32514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32: Bootstrap starting.
I20251026 03:08:19.844851 32514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:19.845270 32514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32: No bootstrap required, opened a new log
I20251026 03:08:19.845366 32514 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b20783b767d4e12896e1583e6cc0a32" member_type: VOTER }
I20251026 03:08:19.845410 32514 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:19.845423 32514 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b20783b767d4e12896e1583e6cc0a32, State: Initialized, Role: FOLLOWER
I20251026 03:08:19.845459 32514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b20783b767d4e12896e1583e6cc0a32" member_type: VOTER }
I20251026 03:08:19.845501 32514 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:08:19.845522 32514 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:08:19.845542 32514 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:19.846005 32514 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b20783b767d4e12896e1583e6cc0a32" member_type: VOTER }
I20251026 03:08:19.846057 32514 leader_election.cc:304] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b20783b767d4e12896e1583e6cc0a32; no voters:
I20251026 03:08:19.846136 32514 leader_election.cc:290] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:08:19.846195 32517 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:19.846261 32514 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:08:19.846333 32517 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 1 LEADER]: Becoming Leader. State: Replica: 1b20783b767d4e12896e1583e6cc0a32, State: Running, Role: LEADER
I20251026 03:08:19.846385 32517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b20783b767d4e12896e1583e6cc0a32" member_type: VOTER }
I20251026 03:08:19.846593 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b20783b767d4e12896e1583e6cc0a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b20783b767d4e12896e1583e6cc0a32" member_type: VOTER } }
I20251026 03:08:19.846622 32519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b20783b767d4e12896e1583e6cc0a32. Latest consensus state: current_term: 1 leader_uuid: "1b20783b767d4e12896e1583e6cc0a32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b20783b767d4e12896e1583e6cc0a32" member_type: VOTER } }
I20251026 03:08:19.846721 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:19.846745 32519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:19.846942 32523 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:08:19.847087 32523 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:08:19.847553 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:08:19.847564 32523 catalog_manager.cc:1357] Generated new cluster ID: 46050144323847b5b1179726db44bbad
I20251026 03:08:19.847646 32523 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:08:19.862732 32523 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:08:19.863140 32523 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:08:19.870471 32523 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32: Generated new TSK 0
I20251026 03:08:19.870539 32523 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:08:19.879344 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:19.880622 32536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.880622 32535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.880688 32538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:19.880781 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:19.880985 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:19.881028 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:19.881048 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448099881048 us; error 0 us; skew 500 ppm
I20251026 03:08:19.881611 31233 webserver.cc:492] Webserver started at http://127.30.128.65:38891/ using document root <none> and password file <none>
I20251026 03:08:19.881695 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:19.881747 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:19.881843 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:19.882139 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.882985 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:19.883402 32543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.883558 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.883599 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.883647 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:19.892105 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:19.892266 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:19.892385 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:19.892550 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:19.892573 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.892604 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:19.892627 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.895309 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:36451
I20251026 03:08:19.895347 32606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:36451 every 8 connection(s)
I20251026 03:08:19.895671 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:19.896696 32612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.896732 32614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.896713 32611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:19.897449 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:19.897562 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:19.897589 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:19.897605 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448099897605 us; error 0 us; skew 500 ppm
I20251026 03:08:19.898192 31233 webserver.cc:492] Webserver started at http://127.30.128.66:38061/ using document root <none> and password file <none>
I20251026 03:08:19.898305 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:19.898352 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:19.898413 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:19.898695 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "0b4cbf03edaa48009bdd7dc46c29301d"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.899029 32607 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33859
I20251026 03:08:19.899092 32607 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:19.899189 32607 heartbeater.cc:507] Master 127.30.128.126:33859 requested a full tablet report, sending...
I20251026 03:08:19.899392 32478 ts_manager.cc:194] Registered new tserver with Master: ba3654b7b5ca4c04bd1c1e9a253e3e6f (127.30.128.65:36451)
I20251026 03:08:19.899691 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:19.899883 32478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49404
I20251026 03:08:19.900116 32619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.900245 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.900290 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "0b4cbf03edaa48009bdd7dc46c29301d"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.900336 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:19.907857 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:19.908027 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:19.908151 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:19.908316 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:19.908340 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.908370 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:19.908394 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.911896 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:33211
I20251026 03:08:19.911932 32682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:33211 every 8 connection(s)
I20251026 03:08:19.912297 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:19.913276 32686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:19.913303 32689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:19.913763 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:19.914233 32687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:19.914403 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:19.914441 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:19.914461 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448099914461 us; error 0 us; skew 500 ppm
I20251026 03:08:19.915082 31233 webserver.cc:492] Webserver started at http://127.30.128.67:43827/ using document root <none> and password file <none>
I20251026 03:08:19.915158 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:19.915202 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:19.915251 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:19.915290 32683 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33859
I20251026 03:08:19.915344 32683 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:19.915438 32683 heartbeater.cc:507] Master 127.30.128.126:33859 requested a full tablet report, sending...
I20251026 03:08:19.915506 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "322d400a370c4b5d93c4fa1aa7195f44"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.915618 32478 ts_manager.cc:194] Registered new tserver with Master: 0b4cbf03edaa48009bdd7dc46c29301d (127.30.128.66:33211)
I20251026 03:08:19.915928 32478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49420
I20251026 03:08:19.916368 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:19.916752 32694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.916869 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.916914 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "322d400a370c4b5d93c4fa1aa7195f44"
format_stamp: "Formatted at 2025-10-26 03:08:19 on dist-test-slave-51kf"
I20251026 03:08:19.916962 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyBasicCalls.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:19.922868 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:19.923043 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:19.923172 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:19.923333 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:19.923357 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.923385 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:19.923410 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:19.926383 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:35789
I20251026 03:08:19.926430 32757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:35789 every 8 connection(s)
I20251026 03:08:19.928917 32758 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33859
I20251026 03:08:19.928961 32758 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:19.929042 32758 heartbeater.cc:507] Master 127.30.128.126:33859 requested a full tablet report, sending...
I20251026 03:08:19.929237 32478 ts_manager.cc:194] Registered new tserver with Master: 322d400a370c4b5d93c4fa1aa7195f44 (127.30.128.67:35789)
I20251026 03:08:19.929543 32478 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49428
I20251026 03:08:19.929826 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003274244s
I20251026 03:08:19.930145 31233 test_util.cc:276] Using random seed: 407461921
I20251026 03:08:19.934871 32478 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49444:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:08:19.935009 32478 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:08:19.939179 32722 tablet_service.cc:1505] Processing CreateTablet for tablet 8bae917eafa744ea972bcc05007833c2 (DEFAULT_TABLE table=test-workload [id=25c80f1c96be4a158a3a78a0ba9abf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:19.939309 32722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bae917eafa744ea972bcc05007833c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:19.939440 32647 tablet_service.cc:1505] Processing CreateTablet for tablet 8bae917eafa744ea972bcc05007833c2 (DEFAULT_TABLE table=test-workload [id=25c80f1c96be4a158a3a78a0ba9abf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:19.939482 32571 tablet_service.cc:1505] Processing CreateTablet for tablet 8bae917eafa744ea972bcc05007833c2 (DEFAULT_TABLE table=test-workload [id=25c80f1c96be4a158a3a78a0ba9abf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:19.939532 32647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bae917eafa744ea972bcc05007833c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:19.939563 32571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bae917eafa744ea972bcc05007833c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:19.940801 305 tablet_bootstrap.cc:492] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: Bootstrap starting.
I20251026 03:08:19.940968 306 tablet_bootstrap.cc:492] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: Bootstrap starting.
I20251026 03:08:19.941087 307 tablet_bootstrap.cc:492] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44: Bootstrap starting.
I20251026 03:08:19.941460 305 tablet_bootstrap.cc:654] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:19.941546 307 tablet_bootstrap.cc:654] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:19.941510 306 tablet_bootstrap.cc:654] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:19.942169 305 tablet_bootstrap.cc:492] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: No bootstrap required, opened a new log
I20251026 03:08:19.942225 305 ts_tablet_manager.cc:1403] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:19.942271 306 tablet_bootstrap.cc:492] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: No bootstrap required, opened a new log
I20251026 03:08:19.942318 306 ts_tablet_manager.cc:1403] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:19.942373 305 raft_consensus.cc:359] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:19.942442 305 raft_consensus.cc:385] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:19.942430 306 raft_consensus.cc:359] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:19.942480 306 raft_consensus.cc:385] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:19.942528 306 raft_consensus.cc:740] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3654b7b5ca4c04bd1c1e9a253e3e6f, State: Initialized, Role: FOLLOWER
I20251026 03:08:19.942574 306 consensus_queue.cc:260] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:19.942649 307 tablet_bootstrap.cc:492] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44: No bootstrap required, opened a new log
I20251026 03:08:19.942699 307 ts_tablet_manager.cc:1403] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 03:08:19.942739 306 ts_tablet_manager.cc:1434] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:08:19.942767 32607 heartbeater.cc:499] Master 127.30.128.126:33859 was elected leader, sending a full tablet report...
I20251026 03:08:19.942474 305 raft_consensus.cc:740] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4cbf03edaa48009bdd7dc46c29301d, State: Initialized, Role: FOLLOWER
I20251026 03:08:19.942837 307 raft_consensus.cc:359] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:19.942848 305 consensus_queue.cc:260] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:19.942906 307 raft_consensus.cc:385] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:19.942931 307 raft_consensus.cc:740] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322d400a370c4b5d93c4fa1aa7195f44, State: Initialized, Role: FOLLOWER
I20251026 03:08:19.942997 307 consensus_queue.cc:260] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:19.943027 305 ts_tablet_manager.cc:1434] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:19.943101 32683 heartbeater.cc:499] Master 127.30.128.126:33859 was elected leader, sending a full tablet report...
I20251026 03:08:19.943157 307 ts_tablet_manager.cc:1434] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:08:19.943259 32758 heartbeater.cc:499] Master 127.30.128.126:33859 was elected leader, sending a full tablet report...
W20251026 03:08:20.145745 32608 tablet.cc:2378] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:08:20.297888 312 raft_consensus.cc:493] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:08:20.298017 312 raft_consensus.cc:515] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:20.298241 312 leader_election.cc:290] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba3654b7b5ca4c04bd1c1e9a253e3e6f (127.30.128.65:36451), 322d400a370c4b5d93c4fa1aa7195f44 (127.30.128.67:35789)
I20251026 03:08:20.300629 32732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2" candidate_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322d400a370c4b5d93c4fa1aa7195f44" is_pre_election: true
I20251026 03:08:20.300724 32732 raft_consensus.cc:2468] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b4cbf03edaa48009bdd7dc46c29301d in term 0.
I20251026 03:08:20.300793 32581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2" candidate_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" is_pre_election: true
I20251026 03:08:20.300868 32581 raft_consensus.cc:2468] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b4cbf03edaa48009bdd7dc46c29301d in term 0.
I20251026 03:08:20.300987 32620 leader_election.cc:304] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4cbf03edaa48009bdd7dc46c29301d, 322d400a370c4b5d93c4fa1aa7195f44; no voters:
I20251026 03:08:20.301085 312 raft_consensus.cc:2804] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:08:20.301129 312 raft_consensus.cc:493] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:08:20.301144 312 raft_consensus.cc:3060] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:20.301801 312 raft_consensus.cc:515] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:20.301923 312 leader_election.cc:290] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [CANDIDATE]: Term 1 election: Requested vote from peers ba3654b7b5ca4c04bd1c1e9a253e3e6f (127.30.128.65:36451), 322d400a370c4b5d93c4fa1aa7195f44 (127.30.128.67:35789)
I20251026 03:08:20.302066 32581 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2" candidate_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f"
I20251026 03:08:20.302066 32732 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2" candidate_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322d400a370c4b5d93c4fa1aa7195f44"
I20251026 03:08:20.302174 32581 raft_consensus.cc:3060] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:20.302181 32732 raft_consensus.cc:3060] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:20.302922 32581 raft_consensus.cc:2468] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4cbf03edaa48009bdd7dc46c29301d in term 1.
I20251026 03:08:20.302927 32732 raft_consensus.cc:2468] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b4cbf03edaa48009bdd7dc46c29301d in term 1.
I20251026 03:08:20.303119 32622 leader_election.cc:304] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b4cbf03edaa48009bdd7dc46c29301d, ba3654b7b5ca4c04bd1c1e9a253e3e6f; no voters:
I20251026 03:08:20.303212 312 raft_consensus.cc:2804] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:20.303337 312 raft_consensus.cc:697] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 LEADER]: Becoming Leader. State: Replica: 0b4cbf03edaa48009bdd7dc46c29301d, State: Running, Role: LEADER
I20251026 03:08:20.303380 312 consensus_queue.cc:237] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:20.303912 32477 catalog_manager.cc:5649] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b4cbf03edaa48009bdd7dc46c29301d (127.30.128.66). New cstate: current_term: 1 leader_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } health_report { overall_health: UNKNOWN } } }
I20251026 03:08:20.386637 32581 raft_consensus.cc:1275] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 1 FOLLOWER]: Refusing update from remote peer 0b4cbf03edaa48009bdd7dc46c29301d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:20.386651 32732 raft_consensus.cc:1275] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 1 FOLLOWER]: Refusing update from remote peer 0b4cbf03edaa48009bdd7dc46c29301d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:20.386902 312 consensus_queue.cc:1048] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:20.386992 315 consensus_queue.cc:1048] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [LEADER]: Connected to new peer: Peer: permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:20.389297 323 mvcc.cc:204] Tried to move back new op lower bound from 7214891419183648768 to 7214891418843103232. Current Snapshot: MvccSnapshot[applied={T|T < 7214891419183648768 or (T in {7214891419183648768})}]
I20251026 03:08:30.995319 354 raft_consensus.cc:993] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: : Instructing follower ba3654b7b5ca4c04bd1c1e9a253e3e6f to start an election
I20251026 03:08:30.995402 354 raft_consensus.cc:1081] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 LEADER]: Signalling peer ba3654b7b5ca4c04bd1c1e9a253e3e6f to start an election
I20251026 03:08:30.995616 32581 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2"
dest_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f"
from {username='slave'} at 127.0.0.1:46486
I20251026 03:08:30.995682 32581 raft_consensus.cc:493] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:08:30.995702 32581 raft_consensus.cc:3060] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:30.996367 32581 raft_consensus.cc:515] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:30.996515 32581 leader_election.cc:290] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [CANDIDATE]: Term 2 election: Requested vote from peers 0b4cbf03edaa48009bdd7dc46c29301d (127.30.128.66:33211), 322d400a370c4b5d93c4fa1aa7195f44 (127.30.128.67:35789)
I20251026 03:08:30.999022 32657 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2" candidate_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14473 } } ignore_live_leader: true dest_uuid: "0b4cbf03edaa48009bdd7dc46c29301d"
I20251026 03:08:30.999094 32657 raft_consensus.cc:3055] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:08:30.999048 32731 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8bae917eafa744ea972bcc05007833c2" candidate_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14473 } } ignore_live_leader: true dest_uuid: "322d400a370c4b5d93c4fa1aa7195f44"
I20251026 03:08:30.999114 32657 raft_consensus.cc:740] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b4cbf03edaa48009bdd7dc46c29301d, State: Running, Role: LEADER
I20251026 03:08:30.999150 32731 raft_consensus.cc:3060] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:30.999190 32657 consensus_queue.cc:260] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14473, Committed index: 14473, Last appended: 1.14473, Last appended by leader: 14473, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:30.999269 32657 raft_consensus.cc:3060] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:30.999783 32731 raft_consensus.cc:2468] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba3654b7b5ca4c04bd1c1e9a253e3e6f in term 2.
I20251026 03:08:30.999850 32657 raft_consensus.cc:2468] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba3654b7b5ca4c04bd1c1e9a253e3e6f in term 2.
I20251026 03:08:30.999979 32544 leader_election.cc:304] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 322d400a370c4b5d93c4fa1aa7195f44, ba3654b7b5ca4c04bd1c1e9a253e3e6f; no voters:
I20251026 03:08:31.000167 360 raft_consensus.cc:2804] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:08:31.000278 360 raft_consensus.cc:697] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 2 LEADER]: Becoming Leader. State: Replica: ba3654b7b5ca4c04bd1c1e9a253e3e6f, State: Running, Role: LEADER
I20251026 03:08:31.000344 360 consensus_queue.cc:237] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14473, Committed index: 14473, Last appended: 1.14473, Last appended by leader: 14473, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } }
I20251026 03:08:31.000761 32472 catalog_manager.cc:5649] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f reported cstate change: term changed from 1 to 2, leader changed from 0b4cbf03edaa48009bdd7dc46c29301d (127.30.128.66) to ba3654b7b5ca4c04bd1c1e9a253e3e6f (127.30.128.65). New cstate: current_term: 2 leader_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba3654b7b5ca4c04bd1c1e9a253e3e6f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 } health_report { overall_health: UNKNOWN } } }
I20251026 03:08:31.007802 32657 raft_consensus.cc:1275] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 2 FOLLOWER]: Refusing update from remote peer ba3654b7b5ca4c04bd1c1e9a253e3e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 14473. Preceding OpId from leader: term: 2 index: 14475. (index mismatch)
I20251026 03:08:31.007802 32731 raft_consensus.cc:1275] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 2 FOLLOWER]: Refusing update from remote peer ba3654b7b5ca4c04bd1c1e9a253e3e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 14473. Preceding OpId from leader: term: 2 index: 14475. (index mismatch)
I20251026 03:08:31.008097 360 consensus_queue.cc:1048] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4cbf03edaa48009bdd7dc46c29301d" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14474, Last known committed idx: 14473, Time since last communication: 0.000s
I20251026 03:08:31.008191 361 consensus_queue.cc:1048] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "322d400a370c4b5d93c4fa1aa7195f44" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14474, Last known committed idx: 14473, Time since last communication: 0.000s
I20251026 03:08:31.011171 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:08:31.013747 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:31.013883 31233 tablet_replica.cc:333] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f: stopping tablet replica
I20251026 03:08:31.013939 31233 raft_consensus.cc:2243] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:08:31.014007 31233 raft_consensus.cc:2272] T 8bae917eafa744ea972bcc05007833c2 P ba3654b7b5ca4c04bd1c1e9a253e3e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:31.017292 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:08:31.018065 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:08:31.021592 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:31.021687 31233 tablet_replica.cc:333] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d: stopping tablet replica
I20251026 03:08:31.021749 31233 raft_consensus.cc:2243] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:31.021816 31233 raft_consensus.cc:2272] T 8bae917eafa744ea972bcc05007833c2 P 0b4cbf03edaa48009bdd7dc46c29301d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:31.042277 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:08:31.043143 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:08:31.046687 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:31.046784 31233 tablet_replica.cc:333] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44: stopping tablet replica
I20251026 03:08:31.046841 31233 raft_consensus.cc:2243] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:31.046887 31233 raft_consensus.cc:2272] T 8bae917eafa744ea972bcc05007833c2 P 322d400a370c4b5d93c4fa1aa7195f44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:31.069134 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:08:31.070039 31233 master.cc:561] Master@127.30.128.126:33859 shutting down...
I20251026 03:08:31.072444 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:31.072517 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:31.072557 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1b20783b767d4e12896e1583e6cc0a32: stopping tablet replica
I20251026 03:08:31.074223 31233 master.cc:583] Master@127.30.128.126:33859 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyBasicCalls (11270 ms)
[ RUN ] TxnParticipantITest.TestBeginCommitAfterFinalize
I20251026 03:08:31.093490 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:42127
I20251026 03:08:31.093659 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:31.095202 377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:31.095184 379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.095170 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:31.095082 376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.095449 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:31.095481 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:31.095494 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448111095494 us; error 0 us; skew 500 ppm
I20251026 03:08:31.096027 31233 webserver.cc:492] Webserver started at http://127.30.128.126:38965/ using document root <none> and password file <none>
I20251026 03:08:31.096135 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:31.096179 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:31.096215 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:31.096468 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "ef1c229ca7bd49acb4c531098bb9dc31"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.097304 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.097733 384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.097955 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.098006 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "ef1c229ca7bd49acb4c531098bb9dc31"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.098049 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:31.105715 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:31.105893 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:31.108184 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:42127
I20251026 03:08:31.108888 436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:42127 every 8 connection(s)
I20251026 03:08:31.109032 437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:31.110080 437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31: Bootstrap starting.
I20251026 03:08:31.110352 437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:31.110787 437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31: No bootstrap required, opened a new log
I20251026 03:08:31.110913 437 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" member_type: VOTER }
I20251026 03:08:31.110960 437 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:31.110982 437 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1c229ca7bd49acb4c531098bb9dc31, State: Initialized, Role: FOLLOWER
I20251026 03:08:31.111038 437 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" member_type: VOTER }
I20251026 03:08:31.111074 437 raft_consensus.cc:399] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:08:31.111109 437 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:08:31.111136 437 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:31.111586 437 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" member_type: VOTER }
I20251026 03:08:31.111639 437 leader_election.cc:304] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef1c229ca7bd49acb4c531098bb9dc31; no voters:
I20251026 03:08:31.111738 437 leader_election.cc:290] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:08:31.111773 440 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:31.111867 437 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:08:31.111886 440 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 1 LEADER]: Becoming Leader. State: Replica: ef1c229ca7bd49acb4c531098bb9dc31, State: Running, Role: LEADER
I20251026 03:08:31.112001 440 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" member_type: VOTER }
I20251026 03:08:31.112197 441 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" member_type: VOTER } }
I20251026 03:08:31.112285 442 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef1c229ca7bd49acb4c531098bb9dc31. Latest consensus state: current_term: 1 leader_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef1c229ca7bd49acb4c531098bb9dc31" member_type: VOTER } }
I20251026 03:08:31.112560 442 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:31.112689 441 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:31.112706 449 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:08:31.112816 449 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:08:31.113391 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:08:31.113426 449 catalog_manager.cc:1357] Generated new cluster ID: 36ca96e1d9714113acd2d520df7c45df
I20251026 03:08:31.113466 449 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:08:31.122876 449 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:08:31.123255 449 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:08:31.128996 449 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31: Generated new TSK 0
I20251026 03:08:31.129065 449 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:08:31.145256 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:31.146499 461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:31.146507 459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.146600 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:31.146512 458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.146878 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:31.146914 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:31.146952 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448111146953 us; error 0 us; skew 500 ppm
I20251026 03:08:31.147500 31233 webserver.cc:492] Webserver started at http://127.30.128.65:37521/ using document root <none> and password file <none>
I20251026 03:08:31.147569 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:31.147605 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:31.147639 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:31.147877 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "9ce1fbdddbb04b8488ee8955427d0a21"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.148684 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.149055 466 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.149189 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.149235 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "9ce1fbdddbb04b8488ee8955427d0a21"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.149271 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:31.181602 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:31.181838 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:31.181993 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:31.182200 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:31.182229 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.182251 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:31.182288 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.184909 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:33467
I20251026 03:08:31.185596 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:08:31.186548 529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:33467 every 8 connection(s)
W20251026 03:08:31.187479 533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:31.187505 532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:31.187479 535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.187740 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:31.187806 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:31.187834 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:31.187860 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448111187860 us; error 0 us; skew 500 ppm
I20251026 03:08:31.188390 31233 webserver.cc:492] Webserver started at http://127.30.128.66:41011/ using document root <none> and password file <none>
I20251026 03:08:31.188467 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:31.188512 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:31.188567 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:31.188820 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "1607fae17aa645609f967cf773144727"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.189843 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.190403 542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.190552 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.190630 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "1607fae17aa645609f967cf773144727"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.190682 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:31.190862 530 heartbeater.cc:344] Connected to a master server at 127.30.128.126:42127
I20251026 03:08:31.190909 530 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:31.191005 530 heartbeater.cc:507] Master 127.30.128.126:42127 requested a full tablet report, sending...
I20251026 03:08:31.191236 401 ts_manager.cc:194] Registered new tserver with Master: 9ce1fbdddbb04b8488ee8955427d0a21 (127.30.128.65:33467)
I20251026 03:08:31.191931 401 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38536
I20251026 03:08:31.205860 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:31.206060 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:31.206207 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:31.206377 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:31.206401 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.206434 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:31.206455 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.209053 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:38529
I20251026 03:08:31.209111 605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:38529 every 8 connection(s)
I20251026 03:08:31.209411 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:31.210516 609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:31.210590 610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.210639 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:31.210983 612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:31.211163 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:31.211196 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:31.211220 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448111211221 us; error 0 us; skew 500 ppm
I20251026 03:08:31.211849 31233 webserver.cc:492] Webserver started at http://127.30.128.67:34739/ using document root <none> and password file <none>
I20251026 03:08:31.211925 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:31.211977 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:31.212033 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:31.212352 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "59f0f0bf40ce42d197333906280b26de"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.212435 606 heartbeater.cc:344] Connected to a master server at 127.30.128.126:42127
I20251026 03:08:31.212498 606 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:31.212600 606 heartbeater.cc:507] Master 127.30.128.126:42127 requested a full tablet report, sending...
I20251026 03:08:31.212814 401 ts_manager.cc:194] Registered new tserver with Master: 1607fae17aa645609f967cf773144727 (127.30.128.66:38529)
I20251026 03:08:31.213147 401 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38538
I20251026 03:08:31.213413 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.213925 617 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.214028 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.214075 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "59f0f0bf40ce42d197333906280b26de"
format_stamp: "Formatted at 2025-10-26 03:08:31 on dist-test-slave-51kf"
I20251026 03:08:31.214130 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestBeginCommitAfterFinalize.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:31.218271 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:31.218465 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:31.218602 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:31.218786 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:31.218812 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.218843 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:31.218869 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.222246 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:40903
I20251026 03:08:31.222308 680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:40903 every 8 connection(s)
I20251026 03:08:31.225119 681 heartbeater.cc:344] Connected to a master server at 127.30.128.126:42127
I20251026 03:08:31.225172 681 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:31.225251 681 heartbeater.cc:507] Master 127.30.128.126:42127 requested a full tablet report, sending...
I20251026 03:08:31.225463 401 ts_manager.cc:194] Registered new tserver with Master: 59f0f0bf40ce42d197333906280b26de (127.30.128.67:40903)
I20251026 03:08:31.225705 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00327927s
I20251026 03:08:31.225802 401 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38548
I20251026 03:08:31.226137 31233 test_util.cc:276] Using random seed: 418757913
I20251026 03:08:31.230064 401 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38562:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:08:31.230185 401 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:08:31.234705 570 tablet_service.cc:1505] Processing CreateTablet for tablet cf59cbbf45f44ae2a2c01038a6ecf82f (DEFAULT_TABLE table=test-workload [id=17393145b36e40369b524b304acfb969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:31.234714 494 tablet_service.cc:1505] Processing CreateTablet for tablet cf59cbbf45f44ae2a2c01038a6ecf82f (DEFAULT_TABLE table=test-workload [id=17393145b36e40369b524b304acfb969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:31.234768 643 tablet_service.cc:1505] Processing CreateTablet for tablet cf59cbbf45f44ae2a2c01038a6ecf82f (DEFAULT_TABLE table=test-workload [id=17393145b36e40369b524b304acfb969]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:31.234871 570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf59cbbf45f44ae2a2c01038a6ecf82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:31.234901 643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf59cbbf45f44ae2a2c01038a6ecf82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:31.234902 494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf59cbbf45f44ae2a2c01038a6ecf82f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:31.236106 696 tablet_bootstrap.cc:492] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: Bootstrap starting.
I20251026 03:08:31.236423 698 tablet_bootstrap.cc:492] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21: Bootstrap starting.
I20251026 03:08:31.236676 696 tablet_bootstrap.cc:654] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:31.236861 698 tablet_bootstrap.cc:654] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:31.237228 697 tablet_bootstrap.cc:492] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de: Bootstrap starting.
I20251026 03:08:31.237345 696 tablet_bootstrap.cc:492] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: No bootstrap required, opened a new log
I20251026 03:08:31.237397 696 ts_tablet_manager.cc:1403] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.237466 698 tablet_bootstrap.cc:492] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21: No bootstrap required, opened a new log
I20251026 03:08:31.237519 698 ts_tablet_manager.cc:1403] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.237553 696 raft_consensus.cc:359] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.237620 696 raft_consensus.cc:385] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:31.237671 697 tablet_bootstrap.cc:654] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:31.237661 696 raft_consensus.cc:740] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1607fae17aa645609f967cf773144727, State: Initialized, Role: FOLLOWER
I20251026 03:08:31.237663 698 raft_consensus.cc:359] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.237793 698 raft_consensus.cc:385] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:31.237818 696 consensus_queue.cc:260] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.237865 698 raft_consensus.cc:740] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce1fbdddbb04b8488ee8955427d0a21, State: Initialized, Role: FOLLOWER
I20251026 03:08:31.237934 698 consensus_queue.cc:260] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.237983 696 ts_tablet_manager.cc:1434] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.238070 606 heartbeater.cc:499] Master 127.30.128.126:42127 was elected leader, sending a full tablet report...
I20251026 03:08:31.238113 698 ts_tablet_manager.cc:1434] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:31.238175 530 heartbeater.cc:499] Master 127.30.128.126:42127 was elected leader, sending a full tablet report...
I20251026 03:08:31.238560 697 tablet_bootstrap.cc:492] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de: No bootstrap required, opened a new log
I20251026 03:08:31.238615 697 ts_tablet_manager.cc:1403] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:31.238744 697 raft_consensus.cc:359] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.238778 697 raft_consensus.cc:385] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:31.238790 697 raft_consensus.cc:740] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59f0f0bf40ce42d197333906280b26de, State: Initialized, Role: FOLLOWER
I20251026 03:08:31.238821 697 consensus_queue.cc:260] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.238926 697 ts_tablet_manager.cc:1434] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:31.238948 681 heartbeater.cc:499] Master 127.30.128.126:42127 was elected leader, sending a full tablet report...
I20251026 03:08:31.282070 701 raft_consensus.cc:493] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:08:31.282193 701 raft_consensus.cc:515] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.282413 701 leader_election.cc:290] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59f0f0bf40ce42d197333906280b26de (127.30.128.67:40903), 9ce1fbdddbb04b8488ee8955427d0a21 (127.30.128.65:33467)
I20251026 03:08:31.285596 504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f" candidate_uuid: "1607fae17aa645609f967cf773144727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" is_pre_election: true
I20251026 03:08:31.285604 655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f" candidate_uuid: "1607fae17aa645609f967cf773144727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f0f0bf40ce42d197333906280b26de" is_pre_election: true
I20251026 03:08:31.285688 504 raft_consensus.cc:2468] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1607fae17aa645609f967cf773144727 in term 0.
I20251026 03:08:31.285715 655 raft_consensus.cc:2468] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1607fae17aa645609f967cf773144727 in term 0.
I20251026 03:08:31.285876 544 leader_election.cc:304] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1607fae17aa645609f967cf773144727, 59f0f0bf40ce42d197333906280b26de; no voters:
I20251026 03:08:31.285996 701 raft_consensus.cc:2804] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:08:31.286036 701 raft_consensus.cc:493] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:08:31.286051 701 raft_consensus.cc:3060] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:31.286625 701 raft_consensus.cc:515] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.286711 701 leader_election.cc:290] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [CANDIDATE]: Term 1 election: Requested vote from peers 59f0f0bf40ce42d197333906280b26de (127.30.128.67:40903), 9ce1fbdddbb04b8488ee8955427d0a21 (127.30.128.65:33467)
I20251026 03:08:31.286877 504 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f" candidate_uuid: "1607fae17aa645609f967cf773144727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce1fbdddbb04b8488ee8955427d0a21"
I20251026 03:08:31.286944 504 raft_consensus.cc:3060] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:31.286875 655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f" candidate_uuid: "1607fae17aa645609f967cf773144727" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59f0f0bf40ce42d197333906280b26de"
I20251026 03:08:31.286978 655 raft_consensus.cc:3060] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:31.287564 655 raft_consensus.cc:2468] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1607fae17aa645609f967cf773144727 in term 1.
I20251026 03:08:31.287573 504 raft_consensus.cc:2468] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1607fae17aa645609f967cf773144727 in term 1.
I20251026 03:08:31.287725 544 leader_election.cc:304] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1607fae17aa645609f967cf773144727, 59f0f0bf40ce42d197333906280b26de; no voters:
I20251026 03:08:31.287799 701 raft_consensus.cc:2804] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:31.287917 701 raft_consensus.cc:697] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 LEADER]: Becoming Leader. State: Replica: 1607fae17aa645609f967cf773144727, State: Running, Role: LEADER
I20251026 03:08:31.287958 701 consensus_queue.cc:237] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:31.288473 400 catalog_manager.cc:5649] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1607fae17aa645609f967cf773144727 (127.30.128.66). New cstate: current_term: 1 leader_uuid: "1607fae17aa645609f967cf773144727" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } health_report { overall_health: UNKNOWN } } }
I20251026 03:08:31.314236 504 raft_consensus.cc:1275] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 1 FOLLOWER]: Refusing update from remote peer 1607fae17aa645609f967cf773144727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:31.314250 655 raft_consensus.cc:1275] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 1 FOLLOWER]: Refusing update from remote peer 1607fae17aa645609f967cf773144727: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:31.314568 706 consensus_queue.cc:1048] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:31.314679 701 consensus_queue.cc:1048] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:41.894694 744 raft_consensus.cc:993] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: : Instructing follower 9ce1fbdddbb04b8488ee8955427d0a21 to start an election
I20251026 03:08:41.894778 744 raft_consensus.cc:1081] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 LEADER]: Signalling peer 9ce1fbdddbb04b8488ee8955427d0a21 to start an election
I20251026 03:08:41.894995 503 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f"
dest_uuid: "9ce1fbdddbb04b8488ee8955427d0a21"
from {username='slave'} at 127.0.0.1:52462
I20251026 03:08:41.895079 503 raft_consensus.cc:493] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:08:41.895102 503 raft_consensus.cc:3060] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:41.895764 503 raft_consensus.cc:515] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:41.895925 503 leader_election.cc:290] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [CANDIDATE]: Term 2 election: Requested vote from peers 59f0f0bf40ce42d197333906280b26de (127.30.128.67:40903), 1607fae17aa645609f967cf773144727 (127.30.128.66:38529)
I20251026 03:08:41.899430 655 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f" candidate_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" candidate_term: 2 candidate_status { last_received { term: 1 index: 14186 } } ignore_live_leader: true dest_uuid: "59f0f0bf40ce42d197333906280b26de"
I20251026 03:08:41.899430 580 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cf59cbbf45f44ae2a2c01038a6ecf82f" candidate_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" candidate_term: 2 candidate_status { last_received { term: 1 index: 14186 } } ignore_live_leader: true dest_uuid: "1607fae17aa645609f967cf773144727"
I20251026 03:08:41.899540 655 raft_consensus.cc:3060] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:41.899549 580 raft_consensus.cc:3055] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:08:41.899598 580 raft_consensus.cc:740] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1607fae17aa645609f967cf773144727, State: Running, Role: LEADER
I20251026 03:08:41.899658 580 consensus_queue.cc:260] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14186, Committed index: 14186, Last appended: 1.14186, Last appended by leader: 14186, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:41.899745 580 raft_consensus.cc:3060] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:41.900343 655 raft_consensus.cc:2468] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ce1fbdddbb04b8488ee8955427d0a21 in term 2.
I20251026 03:08:41.900425 580 raft_consensus.cc:2468] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ce1fbdddbb04b8488ee8955427d0a21 in term 2.
I20251026 03:08:41.900528 468 leader_election.cc:304] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59f0f0bf40ce42d197333906280b26de, 9ce1fbdddbb04b8488ee8955427d0a21; no voters:
I20251026 03:08:41.900708 750 raft_consensus.cc:2804] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:08:41.900847 750 raft_consensus.cc:697] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 2 LEADER]: Becoming Leader. State: Replica: 9ce1fbdddbb04b8488ee8955427d0a21, State: Running, Role: LEADER
I20251026 03:08:41.900923 750 consensus_queue.cc:237] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14186, Committed index: 14186, Last appended: 1.14186, Last appended by leader: 14186, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } }
I20251026 03:08:41.901425 399 catalog_manager.cc:5649] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 reported cstate change: term changed from 1 to 2, leader changed from 1607fae17aa645609f967cf773144727 (127.30.128.66) to 9ce1fbdddbb04b8488ee8955427d0a21 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ce1fbdddbb04b8488ee8955427d0a21" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33467 } health_report { overall_health: HEALTHY } } }
I20251026 03:08:41.905405 655 raft_consensus.cc:1275] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 2 FOLLOWER]: Refusing update from remote peer 9ce1fbdddbb04b8488ee8955427d0a21: Log matching property violated. Preceding OpId in replica: term: 1 index: 14186. Preceding OpId from leader: term: 2 index: 14188. (index mismatch)
I20251026 03:08:41.905405 580 raft_consensus.cc:1275] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 2 FOLLOWER]: Refusing update from remote peer 9ce1fbdddbb04b8488ee8955427d0a21: Log matching property violated. Preceding OpId in replica: term: 1 index: 14186. Preceding OpId from leader: term: 2 index: 14188. (index mismatch)
I20251026 03:08:41.905717 751 consensus_queue.cc:1048] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1607fae17aa645609f967cf773144727" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14187, Last known committed idx: 14186, Time since last communication: 0.000s
I20251026 03:08:41.905827 750 consensus_queue.cc:1048] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59f0f0bf40ce42d197333906280b26de" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 40903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14187, Last known committed idx: 14186, Time since last communication: 0.000s
W20251026 03:08:41.908907 758 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51556: Illegal state: Transaction 0 commit already in progress
I20251026 03:08:41.909181 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:08:41.911680 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:41.911789 31233 tablet_replica.cc:333] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21: stopping tablet replica
I20251026 03:08:41.911840 31233 raft_consensus.cc:2243] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:08:41.911895 31233 raft_consensus.cc:2272] T cf59cbbf45f44ae2a2c01038a6ecf82f P 9ce1fbdddbb04b8488ee8955427d0a21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:41.925650 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:08:41.926746 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:08:41.930186 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:41.930320 31233 tablet_replica.cc:333] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727: stopping tablet replica
I20251026 03:08:41.930393 31233 raft_consensus.cc:2243] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:41.930428 31233 raft_consensus.cc:2272] T cf59cbbf45f44ae2a2c01038a6ecf82f P 1607fae17aa645609f967cf773144727 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:41.950886 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:08:41.951821 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:08:41.954605 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:41.954736 31233 tablet_replica.cc:333] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de: stopping tablet replica
I20251026 03:08:41.954778 31233 raft_consensus.cc:2243] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:41.954819 31233 raft_consensus.cc:2272] T cf59cbbf45f44ae2a2c01038a6ecf82f P 59f0f0bf40ce42d197333906280b26de [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:41.968353 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:08:41.969249 31233 master.cc:561] Master@127.30.128.126:42127 shutting down...
I20251026 03:08:41.971520 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:41.971590 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:41.971609 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P ef1c229ca7bd49acb4c531098bb9dc31: stopping tablet replica
I20251026 03:08:41.984159 31233 master.cc:583] Master@127.30.128.126:42127 shutdown complete.
[ OK ] TxnParticipantITest.TestBeginCommitAfterFinalize (10910 ms)
[ RUN ] TxnParticipantITest.TestProxyErrorWhenNotBegun
I20251026 03:08:42.003903 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:45073
I20251026 03:08:42.004050 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:42.005121 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.005225 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.005219 767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:42.005491 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:42.005622 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:42.005659 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:42.005685 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448122005685 us; error 0 us; skew 500 ppm
I20251026 03:08:42.006310 31233 webserver.cc:492] Webserver started at http://127.30.128.126:36739/ using document root <none> and password file <none>
I20251026 03:08:42.006394 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:42.006441 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:42.006495 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:42.006791 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "61b52a8df1494093a9f57d58a69356b1"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.007748 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:42.008194 774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.008354 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.008402 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "61b52a8df1494093a9f57d58a69356b1"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.008441 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:42.016304 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:42.016462 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:42.019109 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:45073
I20251026 03:08:42.019759 827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:42.027509 826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:45073 every 8 connection(s)
I20251026 03:08:42.030995 827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1: Bootstrap starting.
I20251026 03:08:42.031318 827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:42.031821 827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1: No bootstrap required, opened a new log
I20251026 03:08:42.031955 827 raft_consensus.cc:359] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b52a8df1494093a9f57d58a69356b1" member_type: VOTER }
I20251026 03:08:42.032012 827 raft_consensus.cc:385] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:42.032027 827 raft_consensus.cc:740] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b52a8df1494093a9f57d58a69356b1, State: Initialized, Role: FOLLOWER
I20251026 03:08:42.032081 827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b52a8df1494093a9f57d58a69356b1" member_type: VOTER }
I20251026 03:08:42.032116 827 raft_consensus.cc:399] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:08:42.032136 827 raft_consensus.cc:493] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:08:42.032163 827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:42.032707 827 raft_consensus.cc:515] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b52a8df1494093a9f57d58a69356b1" member_type: VOTER }
I20251026 03:08:42.032763 827 leader_election.cc:304] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61b52a8df1494093a9f57d58a69356b1; no voters:
I20251026 03:08:42.032864 827 leader_election.cc:290] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:08:42.032943 830 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:42.033015 827 sys_catalog.cc:565] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:08:42.033080 830 raft_consensus.cc:697] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 1 LEADER]: Becoming Leader. State: Replica: 61b52a8df1494093a9f57d58a69356b1, State: Running, Role: LEADER
I20251026 03:08:42.033130 830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b52a8df1494093a9f57d58a69356b1" member_type: VOTER }
I20251026 03:08:42.033308 831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61b52a8df1494093a9f57d58a69356b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b52a8df1494093a9f57d58a69356b1" member_type: VOTER } }
I20251026 03:08:42.033366 831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:42.033521 834 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:08:42.033694 834 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:08:42.033713 832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b52a8df1494093a9f57d58a69356b1. Latest consensus state: current_term: 1 leader_uuid: "61b52a8df1494093a9f57d58a69356b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b52a8df1494093a9f57d58a69356b1" member_type: VOTER } }
I20251026 03:08:42.033825 832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:42.034291 834 catalog_manager.cc:1357] Generated new cluster ID: cc23cc1d8f914b898e5e8ec286ddec1e
I20251026 03:08:42.034343 834 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:08:42.034451 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:08:42.053346 834 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:08:42.053843 834 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:08:42.063709 834 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1: Generated new TSK 0
I20251026 03:08:42.063789 834 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:08:42.066242 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:08:42.067631 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:42.067622 848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.067692 849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.067720 851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:42.067853 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:42.067890 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:42.067903 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448122067904 us; error 0 us; skew 500 ppm
I20251026 03:08:42.068480 31233 webserver.cc:492] Webserver started at http://127.30.128.65:46863/ using document root <none> and password file <none>
I20251026 03:08:42.068557 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:42.068603 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:42.068642 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:42.068871 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "1f366de0fb26495b8724be4e65f6bfa4"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.069677 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:42.070160 856 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.070364 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.070415 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "1f366de0fb26495b8724be4e65f6bfa4"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.070457 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:42.081892 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:42.082054 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:42.082170 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:42.082444 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:42.082486 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.082508 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:42.082522 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.085176 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:39785
I20251026 03:08:42.085229 919 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:39785 every 8 connection(s)
I20251026 03:08:42.085534 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:42.086705 924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.086720 925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.086946 927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:42.087031 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:42.087227 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:42.087258 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:42.087270 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448122087271 us; error 0 us; skew 500 ppm
I20251026 03:08:42.088438 31233 webserver.cc:492] Webserver started at http://127.30.128.66:38041/ using document root <none> and password file <none>
I20251026 03:08:42.088536 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:42.088575 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:42.088656 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:42.089038 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "cf2e4657165d4c13b6be8c49c81b5959"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.089089 920 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45073
I20251026 03:08:42.089152 920 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:42.089252 920 heartbeater.cc:507] Master 127.30.128.126:45073 requested a full tablet report, sending...
I20251026 03:08:42.089489 791 ts_manager.cc:194] Registered new tserver with Master: 1f366de0fb26495b8724be4e65f6bfa4 (127.30.128.65:39785)
I20251026 03:08:42.090015 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:42.090166 791 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37954
I20251026 03:08:42.090534 932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.090672 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.090714 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "cf2e4657165d4c13b6be8c49c81b5959"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.090763 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:42.114028 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:42.114235 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:42.114403 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:42.114594 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:42.114620 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.114650 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:42.114674 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.117343 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:34599
I20251026 03:08:42.117415 995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:34599 every 8 connection(s)
I20251026 03:08:42.117782 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:42.118847 999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.118980 1000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:42.119175 1002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:42.119314 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:42.119428 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:42.119460 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:42.119488 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448122119487 us; error 0 us; skew 500 ppm
I20251026 03:08:42.120129 31233 webserver.cc:492] Webserver started at http://127.30.128.67:35775/ using document root <none> and password file <none>
I20251026 03:08:42.120235 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:42.120290 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:42.120337 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:42.120661 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "009fbf064a514844a1810529371d4593"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.120882 996 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45073
I20251026 03:08:42.120929 996 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:42.121028 996 heartbeater.cc:507] Master 127.30.128.126:45073 requested a full tablet report, sending...
I20251026 03:08:42.121294 791 ts_manager.cc:194] Registered new tserver with Master: cf2e4657165d4c13b6be8c49c81b5959 (127.30.128.66:34599)
I20251026 03:08:42.121748 791 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37970
I20251026 03:08:42.121814 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20251026 03:08:42.122380 1007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.122522 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:08:42.122565 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "009fbf064a514844a1810529371d4593"
format_stamp: "Formatted at 2025-10-26 03:08:42 on dist-test-slave-51kf"
I20251026 03:08:42.122606 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:42.134183 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:42.134377 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:42.134526 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:42.134718 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:42.134747 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.134776 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:42.134807 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.137921 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:37893
I20251026 03:08:42.138003 1070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:37893 every 8 connection(s)
I20251026 03:08:42.140745 1071 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45073
I20251026 03:08:42.140799 1071 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:42.140894 1071 heartbeater.cc:507] Master 127.30.128.126:45073 requested a full tablet report, sending...
I20251026 03:08:42.141094 791 ts_manager.cc:194] Registered new tserver with Master: 009fbf064a514844a1810529371d4593 (127.30.128.67:37893)
I20251026 03:08:42.141464 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00334967s
I20251026 03:08:42.141532 791 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37976
I20251026 03:08:42.141891 31233 test_util.cc:276] Using random seed: 429673666
I20251026 03:08:42.146137 791 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37992:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:08:42.146265 791 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:08:42.150811 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 120a071c0d584f0a95c2558f2b195222 (DEFAULT_TABLE table=test-workload [id=287836b72a6f4bb3ba39655c5ddfc05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:42.150952 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120a071c0d584f0a95c2558f2b195222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:42.151090 884 tablet_service.cc:1505] Processing CreateTablet for tablet 120a071c0d584f0a95c2558f2b195222 (DEFAULT_TABLE table=test-workload [id=287836b72a6f4bb3ba39655c5ddfc05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:42.151196 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120a071c0d584f0a95c2558f2b195222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:42.152143 960 tablet_service.cc:1505] Processing CreateTablet for tablet 120a071c0d584f0a95c2558f2b195222 (DEFAULT_TABLE table=test-workload [id=287836b72a6f4bb3ba39655c5ddfc05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:42.152240 960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 120a071c0d584f0a95c2558f2b195222. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:42.152359 1086 tablet_bootstrap.cc:492] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593: Bootstrap starting.
I20251026 03:08:42.152515 1087 tablet_bootstrap.cc:492] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4: Bootstrap starting.
I20251026 03:08:42.152755 1086 tablet_bootstrap.cc:654] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:42.153014 1087 tablet_bootstrap.cc:654] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:42.153338 1086 tablet_bootstrap.cc:492] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593: No bootstrap required, opened a new log
I20251026 03:08:42.153393 1086 ts_tablet_manager.cc:1403] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:42.153574 1090 tablet_bootstrap.cc:492] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: Bootstrap starting.
I20251026 03:08:42.153545 1086 raft_consensus.cc:359] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.153620 1087 tablet_bootstrap.cc:492] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4: No bootstrap required, opened a new log
I20251026 03:08:42.153650 1086 raft_consensus.cc:385] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:42.153672 1086 raft_consensus.cc:740] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009fbf064a514844a1810529371d4593, State: Initialized, Role: FOLLOWER
I20251026 03:08:42.153741 1086 consensus_queue.cc:260] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.153937 1086 ts_tablet_manager.cc:1434] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:42.153664 1087 ts_tablet_manager.cc:1403] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:42.153997 1071 heartbeater.cc:499] Master 127.30.128.126:45073 was elected leader, sending a full tablet report...
I20251026 03:08:42.154107 1090 tablet_bootstrap.cc:654] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:42.154145 1087 raft_consensus.cc:359] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.154213 1087 raft_consensus.cc:385] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:42.154235 1087 raft_consensus.cc:740] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f366de0fb26495b8724be4e65f6bfa4, State: Initialized, Role: FOLLOWER
I20251026 03:08:42.154309 1087 consensus_queue.cc:260] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.154487 1087 ts_tablet_manager.cc:1434] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:42.154536 920 heartbeater.cc:499] Master 127.30.128.126:45073 was elected leader, sending a full tablet report...
I20251026 03:08:42.154798 1090 tablet_bootstrap.cc:492] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: No bootstrap required, opened a new log
I20251026 03:08:42.154847 1090 ts_tablet_manager.cc:1403] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:42.154978 1090 raft_consensus.cc:359] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.155032 1090 raft_consensus.cc:385] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:42.155052 1090 raft_consensus.cc:740] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2e4657165d4c13b6be8c49c81b5959, State: Initialized, Role: FOLLOWER
I20251026 03:08:42.155097 1090 consensus_queue.cc:260] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.155239 1090 ts_tablet_manager.cc:1434] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:08:42.155252 996 heartbeater.cc:499] Master 127.30.128.126:45073 was elected leader, sending a full tablet report...
I20251026 03:08:42.159260 1094 raft_consensus.cc:493] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:08:42.159327 1094 raft_consensus.cc:515] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.159521 1094 leader_election.cc:290] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009fbf064a514844a1810529371d4593 (127.30.128.67:37893), 1f366de0fb26495b8724be4e65f6bfa4 (127.30.128.65:39785)
I20251026 03:08:42.161952 894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222" candidate_uuid: "cf2e4657165d4c13b6be8c49c81b5959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f366de0fb26495b8724be4e65f6bfa4" is_pre_election: true
I20251026 03:08:42.162031 894 raft_consensus.cc:2468] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf2e4657165d4c13b6be8c49c81b5959 in term 0.
I20251026 03:08:42.161957 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222" candidate_uuid: "cf2e4657165d4c13b6be8c49c81b5959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009fbf064a514844a1810529371d4593" is_pre_election: true
I20251026 03:08:42.162079 1045 raft_consensus.cc:2468] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf2e4657165d4c13b6be8c49c81b5959 in term 0.
I20251026 03:08:42.162204 936 leader_election.cc:304] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f366de0fb26495b8724be4e65f6bfa4, cf2e4657165d4c13b6be8c49c81b5959; no voters:
I20251026 03:08:42.162326 1094 raft_consensus.cc:2804] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:08:42.162386 1094 raft_consensus.cc:493] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:08:42.162412 1094 raft_consensus.cc:3060] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:42.162979 1094 raft_consensus.cc:515] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.163177 1094 leader_election.cc:290] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [CANDIDATE]: Term 1 election: Requested vote from peers 009fbf064a514844a1810529371d4593 (127.30.128.67:37893), 1f366de0fb26495b8724be4e65f6bfa4 (127.30.128.65:39785)
I20251026 03:08:42.163290 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222" candidate_uuid: "cf2e4657165d4c13b6be8c49c81b5959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009fbf064a514844a1810529371d4593"
I20251026 03:08:42.163290 894 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222" candidate_uuid: "cf2e4657165d4c13b6be8c49c81b5959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f366de0fb26495b8724be4e65f6bfa4"
I20251026 03:08:42.163357 1045 raft_consensus.cc:3060] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:42.163367 894 raft_consensus.cc:3060] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:42.163926 894 raft_consensus.cc:2468] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2e4657165d4c13b6be8c49c81b5959 in term 1.
I20251026 03:08:42.163987 1045 raft_consensus.cc:2468] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2e4657165d4c13b6be8c49c81b5959 in term 1.
I20251026 03:08:42.164090 936 leader_election.cc:304] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f366de0fb26495b8724be4e65f6bfa4, cf2e4657165d4c13b6be8c49c81b5959; no voters:
I20251026 03:08:42.164171 1094 raft_consensus.cc:2804] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:42.164304 1094 raft_consensus.cc:697] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 LEADER]: Becoming Leader. State: Replica: cf2e4657165d4c13b6be8c49c81b5959, State: Running, Role: LEADER
I20251026 03:08:42.164361 1094 consensus_queue.cc:237] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:42.164835 790 catalog_manager.cc:5649] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf2e4657165d4c13b6be8c49c81b5959 (127.30.128.66). New cstate: current_term: 1 leader_uuid: "cf2e4657165d4c13b6be8c49c81b5959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } health_report { overall_health: HEALTHY } } }
I20251026 03:08:42.175648 1045 raft_consensus.cc:1275] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 1 FOLLOWER]: Refusing update from remote peer cf2e4657165d4c13b6be8c49c81b5959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:42.175662 894 raft_consensus.cc:1275] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 1 FOLLOWER]: Refusing update from remote peer cf2e4657165d4c13b6be8c49c81b5959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:42.175889 1096 consensus_queue.cc:1048] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:42.176069 1096 consensus_queue.cc:1048] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:52.722050 1096 raft_consensus.cc:993] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: : Instructing follower 1f366de0fb26495b8724be4e65f6bfa4 to start an election
I20251026 03:08:52.722141 1096 raft_consensus.cc:1081] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 LEADER]: Signalling peer 1f366de0fb26495b8724be4e65f6bfa4 to start an election
I20251026 03:08:52.722337 894 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222"
dest_uuid: "1f366de0fb26495b8724be4e65f6bfa4"
from {username='slave'} at 127.0.0.1:59830
I20251026 03:08:52.722421 894 raft_consensus.cc:493] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:08:52.722443 894 raft_consensus.cc:3060] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:52.723086 894 raft_consensus.cc:515] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:52.723213 894 leader_election.cc:290] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [CANDIDATE]: Term 2 election: Requested vote from peers 009fbf064a514844a1810529371d4593 (127.30.128.67:37893), cf2e4657165d4c13b6be8c49c81b5959 (127.30.128.66:34599)
I20251026 03:08:52.725648 970 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222" candidate_uuid: "1f366de0fb26495b8724be4e65f6bfa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14342 } } ignore_live_leader: true dest_uuid: "cf2e4657165d4c13b6be8c49c81b5959"
I20251026 03:08:52.725687 1045 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "120a071c0d584f0a95c2558f2b195222" candidate_uuid: "1f366de0fb26495b8724be4e65f6bfa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14342 } } ignore_live_leader: true dest_uuid: "009fbf064a514844a1810529371d4593"
I20251026 03:08:52.725728 970 raft_consensus.cc:3055] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:08:52.725746 1045 raft_consensus.cc:3060] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:52.725759 970 raft_consensus.cc:740] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cf2e4657165d4c13b6be8c49c81b5959, State: Running, Role: LEADER
I20251026 03:08:52.725849 970 consensus_queue.cc:260] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14342, Committed index: 14342, Last appended: 1.14342, Last appended by leader: 14342, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:52.725929 970 raft_consensus.cc:3060] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:08:52.726390 1045 raft_consensus.cc:2468] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f366de0fb26495b8724be4e65f6bfa4 in term 2.
I20251026 03:08:52.726560 859 leader_election.cc:304] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009fbf064a514844a1810529371d4593, 1f366de0fb26495b8724be4e65f6bfa4; no voters:
I20251026 03:08:52.726563 970 raft_consensus.cc:2468] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f366de0fb26495b8724be4e65f6bfa4 in term 2.
I20251026 03:08:52.726750 1150 raft_consensus.cc:2804] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:08:52.726914 1150 raft_consensus.cc:697] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 2 LEADER]: Becoming Leader. State: Replica: 1f366de0fb26495b8724be4e65f6bfa4, State: Running, Role: LEADER
I20251026 03:08:52.726970 1150 consensus_queue.cc:237] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14342, Committed index: 14342, Last appended: 1.14342, Last appended by leader: 14342, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } }
I20251026 03:08:52.727419 784 catalog_manager.cc:5649] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 reported cstate change: term changed from 1 to 2, leader changed from cf2e4657165d4c13b6be8c49c81b5959 (127.30.128.66) to 1f366de0fb26495b8724be4e65f6bfa4 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "1f366de0fb26495b8724be4e65f6bfa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f366de0fb26495b8724be4e65f6bfa4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 } health_report { overall_health: UNKNOWN } } }
W20251026 03:08:52.730376 1158 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49010: Illegal state: Transaction hasn't been successfully started
W20251026 03:08:52.730692 1158 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49010: Illegal state: Transaction metadata for transaction 1 does not exist
I20251026 03:08:52.731143 970 raft_consensus.cc:1275] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 2 FOLLOWER]: Refusing update from remote peer 1f366de0fb26495b8724be4e65f6bfa4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14342. Preceding OpId from leader: term: 2 index: 14344. (index mismatch)
I20251026 03:08:52.731163 1045 raft_consensus.cc:1275] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 2 FOLLOWER]: Refusing update from remote peer 1f366de0fb26495b8724be4e65f6bfa4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14342. Preceding OpId from leader: term: 2 index: 14344. (index mismatch)
I20251026 03:08:52.731297 1151 consensus_queue.cc:1048] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2e4657165d4c13b6be8c49c81b5959" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14343, Last known committed idx: 14342, Time since last communication: 0.000s
I20251026 03:08:52.731359 1150 consensus_queue.cc:1048] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009fbf064a514844a1810529371d4593" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14343, Last known committed idx: 14342, Time since last communication: 0.000s
I20251026 03:08:52.732426 1160 mvcc.cc:204] Tried to move back new op lower bound from 7214891551665930240 to 7214891551650308096. Current Snapshot: MvccSnapshot[applied={T|T < 7214891551665930240}]
I20251026 03:08:52.732776 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:08:52.735652 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:52.735744 31233 tablet_replica.cc:333] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4: stopping tablet replica
I20251026 03:08:52.735792 31233 raft_consensus.cc:2243] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:08:52.735848 31233 raft_consensus.cc:2272] T 120a071c0d584f0a95c2558f2b195222 P 1f366de0fb26495b8724be4e65f6bfa4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:52.749752 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:08:52.750556 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:08:52.753996 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:52.754112 31233 tablet_replica.cc:333] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959: stopping tablet replica
I20251026 03:08:52.754175 31233 raft_consensus.cc:2243] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:52.754228 31233 raft_consensus.cc:2272] T 120a071c0d584f0a95c2558f2b195222 P cf2e4657165d4c13b6be8c49c81b5959 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:52.775290 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:08:52.776185 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:08:52.779651 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:08:52.779794 31233 tablet_replica.cc:333] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593: stopping tablet replica
I20251026 03:08:52.779857 31233 raft_consensus.cc:2243] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:08:52.779984 31233 raft_consensus.cc:2272] T 120a071c0d584f0a95c2558f2b195222 P 009fbf064a514844a1810529371d4593 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:52.793336 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:08:52.794251 31233 master.cc:561] Master@127.30.128.126:45073 shutting down...
I20251026 03:08:52.796789 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:08:52.796860 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:08:52.796880 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 61b52a8df1494093a9f57d58a69356b1: stopping tablet replica
I20251026 03:08:52.808604 31233 master.cc:583] Master@127.30.128.126:45073 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyErrorWhenNotBegun (10823 ms)
[ RUN ] TxnParticipantITest.TestProxyIllegalStatesInCommitSequence
I20251026 03:08:52.827658 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:34217
I20251026 03:08:52.827814 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:08:52.829172 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:52.829233 1165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.829299 1166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.829288 1168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:52.829524 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:52.829558 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:52.829571 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448132829571 us; error 0 us; skew 500 ppm
I20251026 03:08:52.830140 31233 webserver.cc:492] Webserver started at http://127.30.128.126:40417/ using document root <none> and password file <none>
I20251026 03:08:52.830219 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:52.830261 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:52.830305 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:52.830570 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "a2249e35c90940f89fcb9a7059b16983"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.831387 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:52.831799 1173 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.831902 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.831940 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "a2249e35c90940f89fcb9a7059b16983"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.831986 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:52.854285 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:52.854487 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:52.856931 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:34217
I20251026 03:08:52.868121 1225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:34217 every 8 connection(s)
I20251026 03:08:52.868278 1226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:52.869283 1226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983: Bootstrap starting.
I20251026 03:08:52.869563 1226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:52.870086 1226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983: No bootstrap required, opened a new log
I20251026 03:08:52.870206 1226 raft_consensus.cc:359] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e35c90940f89fcb9a7059b16983" member_type: VOTER }
I20251026 03:08:52.870260 1226 raft_consensus.cc:385] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:52.870280 1226 raft_consensus.cc:740] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2249e35c90940f89fcb9a7059b16983, State: Initialized, Role: FOLLOWER
I20251026 03:08:52.870334 1226 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e35c90940f89fcb9a7059b16983" member_type: VOTER }
I20251026 03:08:52.870378 1226 raft_consensus.cc:399] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:08:52.870422 1226 raft_consensus.cc:493] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:08:52.870456 1226 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:52.871070 1226 raft_consensus.cc:515] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e35c90940f89fcb9a7059b16983" member_type: VOTER }
I20251026 03:08:52.871134 1226 leader_election.cc:304] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2249e35c90940f89fcb9a7059b16983; no voters:
I20251026 03:08:52.871256 1226 leader_election.cc:290] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:08:52.871311 1229 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:52.871394 1226 sys_catalog.cc:565] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:08:52.871505 1229 raft_consensus.cc:697] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 1 LEADER]: Becoming Leader. State: Replica: a2249e35c90940f89fcb9a7059b16983, State: Running, Role: LEADER
I20251026 03:08:52.871556 1229 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e35c90940f89fcb9a7059b16983" member_type: VOTER }
I20251026 03:08:52.871765 1230 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2249e35c90940f89fcb9a7059b16983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e35c90940f89fcb9a7059b16983" member_type: VOTER } }
I20251026 03:08:52.871774 1231 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2249e35c90940f89fcb9a7059b16983. Latest consensus state: current_term: 1 leader_uuid: "a2249e35c90940f89fcb9a7059b16983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2249e35c90940f89fcb9a7059b16983" member_type: VOTER } }
I20251026 03:08:52.871834 1230 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:52.871837 1231 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [sys.catalog]: This master's current role is: LEADER
I20251026 03:08:52.872052 1235 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:08:52.872193 1235 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:08:52.872684 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:08:52.872706 1235 catalog_manager.cc:1357] Generated new cluster ID: 1426c8f4fb714188847410760864a224
I20251026 03:08:52.872745 1235 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:08:52.885501 1235 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:08:52.885977 1235 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:08:52.893306 1235 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983: Generated new TSK 0
I20251026 03:08:52.893374 1235 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:08:52.904599 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:08:52.905895 31233 server_base.cc:1047] running on GCE node
W20251026 03:08:52.905885 1247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.905938 1248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.906013 1250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:52.906216 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:52.906248 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:52.906270 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448132906268 us; error 0 us; skew 500 ppm
I20251026 03:08:52.906878 31233 webserver.cc:492] Webserver started at http://127.30.128.65:36319/ using document root <none> and password file <none>
I20251026 03:08:52.906960 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:52.907008 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:52.907058 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:52.907330 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "8428aecd5dc044cfa596d8e2283121ea"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.908195 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:52.908602 1255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.908723 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.908761 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "8428aecd5dc044cfa596d8e2283121ea"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.908807 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:52.941421 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:52.941638 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:52.941823 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:52.942023 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:52.942049 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.942082 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:52.942112 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.944747 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:42035
I20251026 03:08:52.944813 1318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:42035 every 8 connection(s)
I20251026 03:08:52.945118 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:52.946146 1323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.946146 1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.946274 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:52.946470 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:52.946604 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:52.946637 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:52.946653 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448132946654 us; error 0 us; skew 500 ppm
I20251026 03:08:52.947230 31233 webserver.cc:492] Webserver started at http://127.30.128.66:32873/ using document root <none> and password file <none>
I20251026 03:08:52.947315 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:52.947364 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:52.947415 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:52.947744 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "701f4c8b76014788ad41943069ab81a7"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.948709 1319 heartbeater.cc:344] Connected to a master server at 127.30.128.126:34217
I20251026 03:08:52.948777 1319 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:52.948854 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:52.948884 1319 heartbeater.cc:507] Master 127.30.128.126:34217 requested a full tablet report, sending...
I20251026 03:08:52.949100 1190 ts_manager.cc:194] Registered new tserver with Master: 8428aecd5dc044cfa596d8e2283121ea (127.30.128.65:42035)
I20251026 03:08:52.949386 1331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.949494 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.949539 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "701f4c8b76014788ad41943069ab81a7"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.949585 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:52.949815 1190 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35978
I20251026 03:08:52.952971 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:52.953213 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:52.953351 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:52.953517 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:52.953543 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.953579 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:52.953606 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.956207 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:43821
I20251026 03:08:52.956255 1394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:43821 every 8 connection(s)
I20251026 03:08:52.956539 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:08:52.957614 1398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.957633 1399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:08:52.957865 1401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:08:52.958117 31233 server_base.cc:1047] running on GCE node
I20251026 03:08:52.958266 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:08:52.958297 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:08:52.958318 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448132958317 us; error 0 us; skew 500 ppm
I20251026 03:08:52.958940 31233 webserver.cc:492] Webserver started at http://127.30.128.67:36295/ using document root <none> and password file <none>
I20251026 03:08:52.959029 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:08:52.959074 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:08:52.959139 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:08:52.959431 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "5b5257254b6546f1b2e82de9616bafb5"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.959579 1395 heartbeater.cc:344] Connected to a master server at 127.30.128.126:34217
I20251026 03:08:52.959637 1395 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:52.959769 1395 heartbeater.cc:507] Master 127.30.128.126:34217 requested a full tablet report, sending...
I20251026 03:08:52.959987 1190 ts_manager.cc:194] Registered new tserver with Master: 701f4c8b76014788ad41943069ab81a7 (127.30.128.66:43821)
I20251026 03:08:52.960460 1190 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35988
I20251026 03:08:52.960521 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:52.961004 1406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.961108 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.961148 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "5b5257254b6546f1b2e82de9616bafb5"
format_stamp: "Formatted at 2025-10-26 03:08:52 on dist-test-slave-51kf"
I20251026 03:08:52.961180 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:08:52.971959 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:08:52.972126 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:08:52.972249 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:08:52.972416 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:08:52.972441 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.972460 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:08:52.972471 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.975837 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:43057
I20251026 03:08:52.975885 1469 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:43057 every 8 connection(s)
I20251026 03:08:52.978498 1470 heartbeater.cc:344] Connected to a master server at 127.30.128.126:34217
I20251026 03:08:52.978547 1470 heartbeater.cc:461] Registering TS with master...
I20251026 03:08:52.978632 1470 heartbeater.cc:507] Master 127.30.128.126:34217 requested a full tablet report, sending...
I20251026 03:08:52.978825 1190 ts_manager.cc:194] Registered new tserver with Master: 5b5257254b6546f1b2e82de9616bafb5 (127.30.128.67:43057)
I20251026 03:08:52.979125 1190 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35992
I20251026 03:08:52.979296 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003297958s
I20251026 03:08:52.979591 31233 test_util.cc:276] Using random seed: 440511367
I20251026 03:08:52.983888 1190 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35996:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:08:52.984006 1190 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:08:52.988065 1359 tablet_service.cc:1505] Processing CreateTablet for tablet 95de0d0a41934053af25049d1f0d4978 (DEFAULT_TABLE table=test-workload [id=3156888cefb14f5e83829f88711a81a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:52.988183 1359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95de0d0a41934053af25049d1f0d4978. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:52.988437 1434 tablet_service.cc:1505] Processing CreateTablet for tablet 95de0d0a41934053af25049d1f0d4978 (DEFAULT_TABLE table=test-workload [id=3156888cefb14f5e83829f88711a81a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:52.988521 1434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95de0d0a41934053af25049d1f0d4978. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:52.989265 1485 tablet_bootstrap.cc:492] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7: Bootstrap starting.
I20251026 03:08:52.989378 1283 tablet_service.cc:1505] Processing CreateTablet for tablet 95de0d0a41934053af25049d1f0d4978 (DEFAULT_TABLE table=test-workload [id=3156888cefb14f5e83829f88711a81a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:08:52.989471 1283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95de0d0a41934053af25049d1f0d4978. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:08:52.989530 1486 tablet_bootstrap.cc:492] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5: Bootstrap starting.
I20251026 03:08:52.989953 1486 tablet_bootstrap.cc:654] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:52.990482 1486 tablet_bootstrap.cc:492] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5: No bootstrap required, opened a new log
I20251026 03:08:52.990576 1486 ts_tablet_manager.cc:1403] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:08:52.990759 1488 tablet_bootstrap.cc:492] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: Bootstrap starting.
I20251026 03:08:52.990743 1486 raft_consensus.cc:359] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:52.990902 1486 raft_consensus.cc:385] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:52.990945 1486 raft_consensus.cc:740] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b5257254b6546f1b2e82de9616bafb5, State: Initialized, Role: FOLLOWER
I20251026 03:08:52.991029 1486 consensus_queue.cc:260] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:52.991098 1485 tablet_bootstrap.cc:654] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:52.991211 1488 tablet_bootstrap.cc:654] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: Neither blocks nor log segments found. Creating new log.
I20251026 03:08:52.991243 1470 heartbeater.cc:499] Master 127.30.128.126:34217 was elected leader, sending a full tablet report...
I20251026 03:08:52.991231 1486 ts_tablet_manager.cc:1434] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:08:52.991629 1485 tablet_bootstrap.cc:492] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7: No bootstrap required, opened a new log
I20251026 03:08:52.991681 1485 ts_tablet_manager.cc:1403] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:08:52.991813 1485 raft_consensus.cc:359] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:52.991863 1485 raft_consensus.cc:385] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:52.991881 1485 raft_consensus.cc:740] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701f4c8b76014788ad41943069ab81a7, State: Initialized, Role: FOLLOWER
I20251026 03:08:52.991930 1488 tablet_bootstrap.cc:492] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: No bootstrap required, opened a new log
I20251026 03:08:52.991925 1485 consensus_queue.cc:260] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:52.991988 1488 ts_tablet_manager.cc:1403] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:08:52.992082 1485 ts_tablet_manager.cc:1434] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.992105 1395 heartbeater.cc:499] Master 127.30.128.126:34217 was elected leader, sending a full tablet report...
I20251026 03:08:52.992126 1488 raft_consensus.cc:359] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:52.992187 1488 raft_consensus.cc:385] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:08:52.992210 1488 raft_consensus.cc:740] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8428aecd5dc044cfa596d8e2283121ea, State: Initialized, Role: FOLLOWER
I20251026 03:08:52.992251 1488 consensus_queue.cc:260] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:52.992401 1488 ts_tablet_manager.cc:1434] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:08:52.992524 1319 heartbeater.cc:499] Master 127.30.128.126:34217 was elected leader, sending a full tablet report...
W20251026 03:08:53.195132 1320 tablet.cc:2378] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:08:53.216046 1493 raft_consensus.cc:493] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:08:53.216159 1493 raft_consensus.cc:515] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:53.216368 1493 leader_election.cc:290] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b5257254b6546f1b2e82de9616bafb5 (127.30.128.67:43057), 701f4c8b76014788ad41943069ab81a7 (127.30.128.66:43821)
I20251026 03:08:53.219025 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95de0d0a41934053af25049d1f0d4978" candidate_uuid: "8428aecd5dc044cfa596d8e2283121ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701f4c8b76014788ad41943069ab81a7" is_pre_election: true
I20251026 03:08:53.219110 1369 raft_consensus.cc:2468] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8428aecd5dc044cfa596d8e2283121ea in term 0.
I20251026 03:08:53.219141 1444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95de0d0a41934053af25049d1f0d4978" candidate_uuid: "8428aecd5dc044cfa596d8e2283121ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5257254b6546f1b2e82de9616bafb5" is_pre_election: true
I20251026 03:08:53.219233 1444 raft_consensus.cc:2468] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8428aecd5dc044cfa596d8e2283121ea in term 0.
I20251026 03:08:53.219255 1258 leader_election.cc:304] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f4c8b76014788ad41943069ab81a7, 8428aecd5dc044cfa596d8e2283121ea; no voters:
I20251026 03:08:53.219327 1493 raft_consensus.cc:2804] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:08:53.219360 1493 raft_consensus.cc:493] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:08:53.219389 1493 raft_consensus.cc:3060] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:53.220012 1493 raft_consensus.cc:515] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:53.220127 1493 leader_election.cc:290] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [CANDIDATE]: Term 1 election: Requested vote from peers 5b5257254b6546f1b2e82de9616bafb5 (127.30.128.67:43057), 701f4c8b76014788ad41943069ab81a7 (127.30.128.66:43821)
I20251026 03:08:53.220252 1444 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95de0d0a41934053af25049d1f0d4978" candidate_uuid: "8428aecd5dc044cfa596d8e2283121ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b5257254b6546f1b2e82de9616bafb5"
I20251026 03:08:53.220270 1369 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95de0d0a41934053af25049d1f0d4978" candidate_uuid: "8428aecd5dc044cfa596d8e2283121ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "701f4c8b76014788ad41943069ab81a7"
I20251026 03:08:53.220335 1444 raft_consensus.cc:3060] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:53.220345 1369 raft_consensus.cc:3060] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:08:53.221091 1444 raft_consensus.cc:2468] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8428aecd5dc044cfa596d8e2283121ea in term 1.
I20251026 03:08:53.221091 1369 raft_consensus.cc:2468] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8428aecd5dc044cfa596d8e2283121ea in term 1.
I20251026 03:08:53.221279 1258 leader_election.cc:304] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701f4c8b76014788ad41943069ab81a7, 8428aecd5dc044cfa596d8e2283121ea; no voters:
I20251026 03:08:53.221352 1493 raft_consensus.cc:2804] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:08:53.221498 1493 raft_consensus.cc:697] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 1 LEADER]: Becoming Leader. State: Replica: 8428aecd5dc044cfa596d8e2283121ea, State: Running, Role: LEADER
I20251026 03:08:53.221540 1493 consensus_queue.cc:237] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } }
I20251026 03:08:53.222056 1189 catalog_manager.cc:5649] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 8428aecd5dc044cfa596d8e2283121ea (127.30.128.65). New cstate: current_term: 1 leader_uuid: "8428aecd5dc044cfa596d8e2283121ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8428aecd5dc044cfa596d8e2283121ea" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 42035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 } health_report { overall_health: UNKNOWN } } }
I20251026 03:08:53.274909 1444 raft_consensus.cc:1275] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 1 FOLLOWER]: Refusing update from remote peer 8428aecd5dc044cfa596d8e2283121ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:53.274909 1369 raft_consensus.cc:1275] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 1 FOLLOWER]: Refusing update from remote peer 8428aecd5dc044cfa596d8e2283121ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:08:53.275159 1495 consensus_queue.cc:1048] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b5257254b6546f1b2e82de9616bafb5" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 43057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:08:53.275318 1493 consensus_queue.cc:1048] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "701f4c8b76014788ad41943069ab81a7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 03:09:03.279090 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction 0 already open
W20251026 03:09:03.279428 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Cannot finalize transaction in state: OPEN
W20251026 03:09:03.280452 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction 0 commit already in progress
W20251026 03:09:03.280710 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:09:03.281862 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction 0 commit already in progress
W20251026 03:09:03.282183 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction metadata for transaction 0 already set
W20251026 03:09:03.282451 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:09:03.282714 1502 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51310: Illegal state: Transaction metadata for transaction 0 already exists
I20251026 03:09:03.282909 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:09:03.285058 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:03.285162 31233 tablet_replica.cc:333] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea: stopping tablet replica
I20251026 03:09:03.285223 31233 raft_consensus.cc:2243] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:03.285264 31233 raft_consensus.cc:2272] T 95de0d0a41934053af25049d1f0d4978 P 8428aecd5dc044cfa596d8e2283121ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:03.298945 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:09:03.299760 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:09:03.301967 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:03.302073 31233 tablet_replica.cc:333] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7: stopping tablet replica
I20251026 03:09:03.302147 31233 raft_consensus.cc:2243] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:03.302189 31233 raft_consensus.cc:2272] T 95de0d0a41934053af25049d1f0d4978 P 701f4c8b76014788ad41943069ab81a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:03.312063 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:09:03.312698 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:09:03.315461 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:03.315548 31233 tablet_replica.cc:333] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5: stopping tablet replica
I20251026 03:09:03.315596 31233 raft_consensus.cc:2243] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:03.315645 31233 raft_consensus.cc:2272] T 95de0d0a41934053af25049d1f0d4978 P 5b5257254b6546f1b2e82de9616bafb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:03.319268 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:09:03.320029 31233 master.cc:561] Master@127.30.128.126:34217 shutting down...
I20251026 03:09:03.325040 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:03.325100 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:03.325136 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P a2249e35c90940f89fcb9a7059b16983: stopping tablet replica
I20251026 03:09:03.336668 31233 master.cc:583] Master@127.30.128.126:34217 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyIllegalStatesInCommitSequence (10529 ms)
[ RUN ] TxnParticipantITest.TestProxyIllegalStatesInAbortSequence
I20251026 03:09:03.357514 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:33817
I20251026 03:09:03.357683 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:03.359145 1544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:03.359158 1545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:03.359171 1547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:03.359406 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:03.359527 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:03.359568 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:03.359580 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448143359581 us; error 0 us; skew 500 ppm
I20251026 03:09:03.360105 31233 webserver.cc:492] Webserver started at http://127.30.128.126:45267/ using document root <none> and password file <none>
I20251026 03:09:03.360162 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:03.360189 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:03.360222 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:03.360431 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "e7c89d1c9a35426da3230e7ad45f3dd4"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.361218 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:03.361604 1552 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.361900 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.361945 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "e7c89d1c9a35426da3230e7ad45f3dd4"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.361980 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:03.371572 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:03.371737 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:03.373946 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:33817
I20251026 03:09:03.381949 1604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:33817 every 8 connection(s)
I20251026 03:09:03.385002 1605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:03.386044 1605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4: Bootstrap starting.
I20251026 03:09:03.386312 1605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:03.386754 1605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4: No bootstrap required, opened a new log
I20251026 03:09:03.386858 1605 raft_consensus.cc:359] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" member_type: VOTER }
I20251026 03:09:03.386901 1605 raft_consensus.cc:385] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:03.386914 1605 raft_consensus.cc:740] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c89d1c9a35426da3230e7ad45f3dd4, State: Initialized, Role: FOLLOWER
I20251026 03:09:03.386950 1605 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" member_type: VOTER }
I20251026 03:09:03.386991 1605 raft_consensus.cc:399] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:09:03.387008 1605 raft_consensus.cc:493] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:09:03.387027 1605 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:03.387451 1605 raft_consensus.cc:515] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" member_type: VOTER }
I20251026 03:09:03.387501 1605 leader_election.cc:304] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7c89d1c9a35426da3230e7ad45f3dd4; no voters:
I20251026 03:09:03.387580 1605 leader_election.cc:290] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:09:03.387637 1608 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:03.387707 1605 sys_catalog.cc:565] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:09:03.387775 1608 raft_consensus.cc:697] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 1 LEADER]: Becoming Leader. State: Replica: e7c89d1c9a35426da3230e7ad45f3dd4, State: Running, Role: LEADER
I20251026 03:09:03.387831 1608 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" member_type: VOTER }
I20251026 03:09:03.388070 1610 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c89d1c9a35426da3230e7ad45f3dd4. Latest consensus state: current_term: 1 leader_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" member_type: VOTER } }
I20251026 03:09:03.388150 1610 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:03.388087 1609 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7c89d1c9a35426da3230e7ad45f3dd4" member_type: VOTER } }
I20251026 03:09:03.388286 1609 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:03.388437 1614 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:09:03.388576 1614 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:09:03.389077 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:09:03.389168 1614 catalog_manager.cc:1357] Generated new cluster ID: 2e94cac77ef04a47a35994e30facda4e
I20251026 03:09:03.389214 1614 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:09:03.399223 1614 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:09:03.399646 1614 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:09:03.410405 1614 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4: Generated new TSK 0
I20251026 03:09:03.410477 1614 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:09:03.421020 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:03.422362 1626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:03.422441 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:03.422467 1627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:03.422523 1629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:03.422636 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:03.422683 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:03.422701 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448143422702 us; error 0 us; skew 500 ppm
I20251026 03:09:03.423213 31233 webserver.cc:492] Webserver started at http://127.30.128.65:42135/ using document root <none> and password file <none>
I20251026 03:09:03.423297 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:03.423329 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:03.423363 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:03.423570 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "c054e3f2818b4c6d855d4f3e4e193550"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.424335 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:03.424712 1634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.424815 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.424856 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "c054e3f2818b4c6d855d4f3e4e193550"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.424890 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:03.434005 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:03.434151 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:03.434271 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:03.434435 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:03.434461 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.434479 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:03.434505 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.436950 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:35847
I20251026 03:09:03.437005 1697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:35847 every 8 connection(s)
I20251026 03:09:03.437304 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:03.438251 1702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:03.438297 1705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:03.438483 1703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:03.438684 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:03.438781 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:03.438799 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:03.438812 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448143438811 us; error 0 us; skew 500 ppm
I20251026 03:09:03.439328 31233 webserver.cc:492] Webserver started at http://127.30.128.66:37979/ using document root <none> and password file <none>
I20251026 03:09:03.439411 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:03.439456 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:03.439500 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:03.439846 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "14337dded7d647c4a1d9101ac21c766b"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.440654 1698 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33817
I20251026 03:09:03.440732 1698 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:03.440825 1698 heartbeater.cc:507] Master 127.30.128.126:33817 requested a full tablet report, sending...
I20251026 03:09:03.440865 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 03:09:03.441045 1569 ts_manager.cc:194] Registered new tserver with Master: c054e3f2818b4c6d855d4f3e4e193550 (127.30.128.65:35847)
I20251026 03:09:03.441329 1710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.441452 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.441498 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "14337dded7d647c4a1d9101ac21c766b"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.441536 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:03.441895 1569 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55078
I20251026 03:09:03.448005 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:03.448134 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:03.448237 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:03.448418 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:03.448446 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.448465 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:03.448489 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.451120 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:41629
I20251026 03:09:03.451181 1774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:41629 every 8 connection(s)
I20251026 03:09:03.451460 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:03.452387 1779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:03.452629 1778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:03.452761 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:03.452461 1781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:03.453078 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:03.453114 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:03.453135 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448143453133 us; error 0 us; skew 500 ppm
I20251026 03:09:03.453706 31233 webserver.cc:492] Webserver started at http://127.30.128.67:34571/ using document root <none> and password file <none>
I20251026 03:09:03.453815 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:03.453869 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:03.453920 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:03.454210 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "ab79e0d95ed242319651b81070c85d94"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.454552 1775 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33817
I20251026 03:09:03.454608 1775 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:03.454695 1775 heartbeater.cc:507] Master 127.30.128.126:33817 requested a full tablet report, sending...
I20251026 03:09:03.454923 1569 ts_manager.cc:194] Registered new tserver with Master: 14337dded7d647c4a1d9101ac21c766b (127.30.128.66:41629)
I20251026 03:09:03.455210 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:03.455361 1569 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55094
I20251026 03:09:03.455672 1786 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.455821 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.455860 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "ab79e0d95ed242319651b81070c85d94"
format_stamp: "Formatted at 2025-10-26 03:09:03 on dist-test-slave-51kf"
I20251026 03:09:03.455912 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyIllegalStatesInAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:03.467237 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:03.467406 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:03.467540 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:03.467717 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:03.467744 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.467775 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:03.467801 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:03.470920 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:34895
I20251026 03:09:03.470978 1849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:34895 every 8 connection(s)
I20251026 03:09:03.473662 1850 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33817
I20251026 03:09:03.473711 1850 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:03.473820 1850 heartbeater.cc:507] Master 127.30.128.126:33817 requested a full tablet report, sending...
I20251026 03:09:03.474040 1569 ts_manager.cc:194] Registered new tserver with Master: ab79e0d95ed242319651b81070c85d94 (127.30.128.67:34895)
I20251026 03:09:03.474376 1569 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55098
I20251026 03:09:03.474381 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003304064s
I20251026 03:09:03.474732 31233 test_util.cc:276] Using random seed: 451006508
I20251026 03:09:03.478717 1569 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55110:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:09:03.478839 1569 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:09:03.483036 1662 tablet_service.cc:1505] Processing CreateTablet for tablet 3f29538b15454911b3694ca598048575 (DEFAULT_TABLE table=test-workload [id=5a42bc1ac10d4e0dad41d7c981712c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:03.483156 1662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f29538b15454911b3694ca598048575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:03.483762 1738 tablet_service.cc:1505] Processing CreateTablet for tablet 3f29538b15454911b3694ca598048575 (DEFAULT_TABLE table=test-workload [id=5a42bc1ac10d4e0dad41d7c981712c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:03.483815 1814 tablet_service.cc:1505] Processing CreateTablet for tablet 3f29538b15454911b3694ca598048575 (DEFAULT_TABLE table=test-workload [id=5a42bc1ac10d4e0dad41d7c981712c89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:03.483873 1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f29538b15454911b3694ca598048575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:03.483927 1814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f29538b15454911b3694ca598048575. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:03.484820 1865 tablet_bootstrap.cc:492] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550: Bootstrap starting.
I20251026 03:09:03.485159 1866 tablet_bootstrap.cc:492] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b: Bootstrap starting.
I20251026 03:09:03.485285 1865 tablet_bootstrap.cc:654] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:03.485440 1867 tablet_bootstrap.cc:492] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94: Bootstrap starting.
I20251026 03:09:03.485743 1866 tablet_bootstrap.cc:654] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:03.485869 1867 tablet_bootstrap.cc:654] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:03.486126 1865 tablet_bootstrap.cc:492] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550: No bootstrap required, opened a new log
I20251026 03:09:03.486186 1865 ts_tablet_manager.cc:1403] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:03.486337 1866 tablet_bootstrap.cc:492] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b: No bootstrap required, opened a new log
I20251026 03:09:03.486398 1866 ts_tablet_manager.cc:1403] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:03.486341 1865 raft_consensus.cc:359] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.486471 1865 raft_consensus.cc:385] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:03.486497 1865 raft_consensus.cc:740] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c054e3f2818b4c6d855d4f3e4e193550, State: Initialized, Role: FOLLOWER
I20251026 03:09:03.486545 1865 consensus_queue.cc:260] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.486574 1867 tablet_bootstrap.cc:492] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94: No bootstrap required, opened a new log
I20251026 03:09:03.486544 1866 raft_consensus.cc:359] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.486627 1867 ts_tablet_manager.cc:1403] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:03.486665 1866 raft_consensus.cc:385] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:03.486694 1866 raft_consensus.cc:740] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14337dded7d647c4a1d9101ac21c766b, State: Initialized, Role: FOLLOWER
I20251026 03:09:03.486711 1865 ts_tablet_manager.cc:1434] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:09:03.486763 1866 consensus_queue.cc:260] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.486801 1698 heartbeater.cc:499] Master 127.30.128.126:33817 was elected leader, sending a full tablet report...
I20251026 03:09:03.486788 1867 raft_consensus.cc:359] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.486941 1775 heartbeater.cc:499] Master 127.30.128.126:33817 was elected leader, sending a full tablet report...
I20251026 03:09:03.486927 1866 ts_tablet_manager.cc:1434] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:09:03.486976 1867 raft_consensus.cc:385] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:03.487008 1867 raft_consensus.cc:740] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab79e0d95ed242319651b81070c85d94, State: Initialized, Role: FOLLOWER
I20251026 03:09:03.487052 1867 consensus_queue.cc:260] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.487195 1867 ts_tablet_manager.cc:1434] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:09:03.487264 1850 heartbeater.cc:499] Master 127.30.128.126:33817 was elected leader, sending a full tablet report...
I20251026 03:09:03.537251 1871 raft_consensus.cc:493] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:09:03.537355 1871 raft_consensus.cc:515] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.537524 1871 leader_election.cc:290] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab79e0d95ed242319651b81070c85d94 (127.30.128.67:34895), 14337dded7d647c4a1d9101ac21c766b (127.30.128.66:41629)
I20251026 03:09:03.540043 1748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f29538b15454911b3694ca598048575" candidate_uuid: "c054e3f2818b4c6d855d4f3e4e193550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14337dded7d647c4a1d9101ac21c766b" is_pre_election: true
I20251026 03:09:03.540108 1824 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f29538b15454911b3694ca598048575" candidate_uuid: "c054e3f2818b4c6d855d4f3e4e193550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab79e0d95ed242319651b81070c85d94" is_pre_election: true
I20251026 03:09:03.540138 1748 raft_consensus.cc:2468] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c054e3f2818b4c6d855d4f3e4e193550 in term 0.
I20251026 03:09:03.540149 1824 raft_consensus.cc:2468] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c054e3f2818b4c6d855d4f3e4e193550 in term 0.
I20251026 03:09:03.540277 1636 leader_election.cc:304] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14337dded7d647c4a1d9101ac21c766b, c054e3f2818b4c6d855d4f3e4e193550; no voters:
I20251026 03:09:03.540393 1871 raft_consensus.cc:2804] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:09:03.540438 1871 raft_consensus.cc:493] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:09:03.540469 1871 raft_consensus.cc:3060] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:03.541081 1871 raft_consensus.cc:515] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.541203 1871 leader_election.cc:290] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [CANDIDATE]: Term 1 election: Requested vote from peers ab79e0d95ed242319651b81070c85d94 (127.30.128.67:34895), 14337dded7d647c4a1d9101ac21c766b (127.30.128.66:41629)
I20251026 03:09:03.541313 1824 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f29538b15454911b3694ca598048575" candidate_uuid: "c054e3f2818b4c6d855d4f3e4e193550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab79e0d95ed242319651b81070c85d94"
I20251026 03:09:03.541323 1748 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f29538b15454911b3694ca598048575" candidate_uuid: "c054e3f2818b4c6d855d4f3e4e193550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14337dded7d647c4a1d9101ac21c766b"
I20251026 03:09:03.541383 1824 raft_consensus.cc:3060] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:03.541391 1748 raft_consensus.cc:3060] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:03.542172 1748 raft_consensus.cc:2468] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c054e3f2818b4c6d855d4f3e4e193550 in term 1.
I20251026 03:09:03.542172 1824 raft_consensus.cc:2468] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c054e3f2818b4c6d855d4f3e4e193550 in term 1.
I20251026 03:09:03.542361 1638 leader_election.cc:304] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab79e0d95ed242319651b81070c85d94, c054e3f2818b4c6d855d4f3e4e193550; no voters:
I20251026 03:09:03.542480 1871 raft_consensus.cc:2804] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:03.542619 1871 raft_consensus.cc:697] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 1 LEADER]: Becoming Leader. State: Replica: c054e3f2818b4c6d855d4f3e4e193550, State: Running, Role: LEADER
I20251026 03:09:03.542673 1871 consensus_queue.cc:237] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } }
I20251026 03:09:03.543208 1568 catalog_manager.cc:5649] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 reported cstate change: term changed from 0 to 1, leader changed from <none> to c054e3f2818b4c6d855d4f3e4e193550 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "c054e3f2818b4c6d855d4f3e4e193550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c054e3f2818b4c6d855d4f3e4e193550" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 } health_report { overall_health: UNKNOWN } } }
I20251026 03:09:03.564985 1824 raft_consensus.cc:1275] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 1 FOLLOWER]: Refusing update from remote peer c054e3f2818b4c6d855d4f3e4e193550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:03.564985 1748 raft_consensus.cc:1275] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 1 FOLLOWER]: Refusing update from remote peer c054e3f2818b4c6d855d4f3e4e193550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:03.565236 1875 consensus_queue.cc:1048] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14337dded7d647c4a1d9101ac21c766b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:03.565325 1871 consensus_queue.cc:1048] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab79e0d95ed242319651b81070c85d94" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 03:09:13.564951 1882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50724: Illegal state: Transaction 0 already open
W20251026 03:09:13.565279 1882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50724: Illegal state: Cannot finalize transaction in state: OPEN
W20251026 03:09:13.566412 1882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50724: Illegal state: Transaction metadata for transaction 0 already set
W20251026 03:09:13.566706 1882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50724: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:09:13.566975 1882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50724: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:09:13.567242 1882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50724: Illegal state: Cannot begin committing transaction in state: INITIALIZING
I20251026 03:09:13.567396 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:09:13.569815 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:13.569939 31233 tablet_replica.cc:333] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550: stopping tablet replica
I20251026 03:09:13.570000 31233 raft_consensus.cc:2243] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:13.570080 31233 raft_consensus.cc:2272] T 3f29538b15454911b3694ca598048575 P c054e3f2818b4c6d855d4f3e4e193550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:13.584252 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:09:13.585215 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:09:13.587790 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:13.587888 31233 tablet_replica.cc:333] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b: stopping tablet replica
I20251026 03:09:13.587941 31233 raft_consensus.cc:2243] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:13.587993 31233 raft_consensus.cc:2272] T 3f29538b15454911b3694ca598048575 P 14337dded7d647c4a1d9101ac21c766b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:13.608132 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:09:13.609025 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:09:13.611799 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:13.611912 31233 tablet_replica.cc:333] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94: stopping tablet replica
I20251026 03:09:13.611971 31233 raft_consensus.cc:2243] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:13.612020 31233 raft_consensus.cc:2272] T 3f29538b15454911b3694ca598048575 P ab79e0d95ed242319651b81070c85d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:13.625433 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:09:13.626405 31233 master.cc:561] Master@127.30.128.126:33817 shutting down...
I20251026 03:09:13.628681 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:13.628758 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:13.628788 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P e7c89d1c9a35426da3230e7ad45f3dd4: stopping tablet replica
I20251026 03:09:13.640549 31233 master.cc:583] Master@127.30.128.126:33817 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyIllegalStatesInAbortSequence (10301 ms)
[ RUN ] TxnParticipantITest.TestProxyNonLeader
I20251026 03:09:13.659395 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:44977
I20251026 03:09:13.659575 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:13.660729 1926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.660730 1923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.660812 1924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:13.660900 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:13.661130 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:13.661160 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:13.661172 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448153661173 us; error 0 us; skew 500 ppm
I20251026 03:09:13.661671 31233 webserver.cc:492] Webserver started at http://127.30.128.126:43151/ using document root <none> and password file <none>
I20251026 03:09:13.661743 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:13.661823 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:13.661873 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:13.662144 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "cfd5eb48f88c4a30af1fc1ebc888f232"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.662943 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:13.663371 1931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.663496 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.663534 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "cfd5eb48f88c4a30af1fc1ebc888f232"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.663584 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:13.673609 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:13.673863 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:13.676195 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:44977
I20251026 03:09:13.676446 1983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:44977 every 8 connection(s)
I20251026 03:09:13.677260 1984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:13.688319 1984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232: Bootstrap starting.
I20251026 03:09:13.688608 1984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:13.689095 1984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232: No bootstrap required, opened a new log
I20251026 03:09:13.689204 1984 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" member_type: VOTER }
I20251026 03:09:13.689249 1984 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:13.689262 1984 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd5eb48f88c4a30af1fc1ebc888f232, State: Initialized, Role: FOLLOWER
I20251026 03:09:13.689294 1984 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" member_type: VOTER }
I20251026 03:09:13.689323 1984 raft_consensus.cc:399] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:09:13.689342 1984 raft_consensus.cc:493] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:09:13.689364 1984 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:13.689883 1984 raft_consensus.cc:515] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" member_type: VOTER }
I20251026 03:09:13.689940 1984 leader_election.cc:304] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd5eb48f88c4a30af1fc1ebc888f232; no voters:
I20251026 03:09:13.690058 1984 leader_election.cc:290] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:09:13.690119 1987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:13.690212 1984 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:09:13.690305 1987 raft_consensus.cc:697] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 1 LEADER]: Becoming Leader. State: Replica: cfd5eb48f88c4a30af1fc1ebc888f232, State: Running, Role: LEADER
I20251026 03:09:13.690367 1987 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" member_type: VOTER }
I20251026 03:09:13.690601 1989 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd5eb48f88c4a30af1fc1ebc888f232. Latest consensus state: current_term: 1 leader_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" member_type: VOTER } }
I20251026 03:09:13.690780 1989 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:13.690590 1988 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5eb48f88c4a30af1fc1ebc888f232" member_type: VOTER } }
I20251026 03:09:13.690943 1988 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:13.691025 1996 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:09:13.691141 1996 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:09:13.691584 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:09:13.691609 1996 catalog_manager.cc:1357] Generated new cluster ID: d646fea798dc4cd0bf65b2d0409ac3cd
I20251026 03:09:13.691637 1996 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:09:13.713616 1996 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:09:13.714047 1996 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:09:13.720625 1996 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232: Generated new TSK 0
I20251026 03:09:13.720705 1996 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:09:13.723305 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:09:13.724618 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:13.724606 2005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.724740 2006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.724740 2008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:13.724960 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:13.724993 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:13.725005 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448153725006 us; error 0 us; skew 500 ppm
I20251026 03:09:13.725518 31233 webserver.cc:492] Webserver started at http://127.30.128.65:41991/ using document root <none> and password file <none>
I20251026 03:09:13.725605 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:13.725639 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:13.725673 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:13.725924 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.726722 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:13.727111 2013 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.727228 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.727272 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.727308 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:13.734040 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:13.734200 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:13.734313 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:13.734483 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:13.734508 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.734527 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:13.734552 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.737182 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:39611
I20251026 03:09:13.737244 2076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:39611 every 8 connection(s)
I20251026 03:09:13.737543 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:13.738898 2081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.739094 2084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:13.739423 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:13.739799 2082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:13.740005 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:13.740056 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:13.740075 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448153740075 us; error 0 us; skew 500 ppm
I20251026 03:09:13.740983 31233 webserver.cc:492] Webserver started at http://127.30.128.66:43409/ using document root <none> and password file <none>
I20251026 03:09:13.741079 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:13.741132 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:13.741149 2077 heartbeater.cc:344] Connected to a master server at 127.30.128.126:44977
I20251026 03:09:13.741183 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:13.741227 2077 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:13.741348 2077 heartbeater.cc:507] Master 127.30.128.126:44977 requested a full tablet report, sending...
I20251026 03:09:13.741528 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "06765e025bc9477d8f98b95bb9f69a6b"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.741560 1948 ts_manager.cc:194] Registered new tserver with Master: 2e6b9bc02bab4fb7ac5d8dea9b152e93 (127.30.128.65:39611)
I20251026 03:09:13.742163 1948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42982
I20251026 03:09:13.742523 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:13.743053 2089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.743170 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:09:13.743216 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "06765e025bc9477d8f98b95bb9f69a6b"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.743261 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:13.748484 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:13.748656 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:13.748795 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:13.748976 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:13.749001 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.749039 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:13.749063 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.751796 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:40787
I20251026 03:09:13.751852 2152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:40787 every 8 connection(s)
I20251026 03:09:13.752171 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:13.753202 2156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.753291 2157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:13.753386 2159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:13.753427 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:13.753647 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:13.753690 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:13.753715 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448153753715 us; error 0 us; skew 500 ppm
I20251026 03:09:13.754395 31233 webserver.cc:492] Webserver started at http://127.30.128.67:39407/ using document root <none> and password file <none>
I20251026 03:09:13.754475 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:13.754520 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:13.754566 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:13.754885 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "b631b6f2ebc043cf9a2b197c30664e3c"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.755076 2153 heartbeater.cc:344] Connected to a master server at 127.30.128.126:44977
I20251026 03:09:13.755125 2153 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:13.755228 2153 heartbeater.cc:507] Master 127.30.128.126:44977 requested a full tablet report, sending...
I20251026 03:09:13.755422 1948 ts_manager.cc:194] Registered new tserver with Master: 06765e025bc9477d8f98b95bb9f69a6b (127.30.128.66:40787)
I20251026 03:09:13.755731 1948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42996
I20251026 03:09:13.755847 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:13.756253 2164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.756371 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.756415 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "b631b6f2ebc043cf9a2b197c30664e3c"
format_stamp: "Formatted at 2025-10-26 03:09:13 on dist-test-slave-51kf"
I20251026 03:09:13.756461 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyNonLeader.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:13.764994 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:13.765177 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:13.765323 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:13.765508 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:13.765537 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.765566 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:13.765590 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.768558 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:33521
I20251026 03:09:13.768594 2227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:33521 every 8 connection(s)
I20251026 03:09:13.771507 2228 heartbeater.cc:344] Connected to a master server at 127.30.128.126:44977
I20251026 03:09:13.771579 2228 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:13.771688 2228 heartbeater.cc:507] Master 127.30.128.126:44977 requested a full tablet report, sending...
I20251026 03:09:13.771924 1948 ts_manager.cc:194] Registered new tserver with Master: b631b6f2ebc043cf9a2b197c30664e3c (127.30.128.67:33521)
I20251026 03:09:13.772001 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003270221s
I20251026 03:09:13.772368 31233 test_util.cc:276] Using random seed: 461304143
I20251026 03:09:13.772387 1948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43004
I20251026 03:09:13.776693 1948 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43018:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:09:13.776823 1948 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:09:13.781209 2117 tablet_service.cc:1505] Processing CreateTablet for tablet 04dd7fdbeb794b92865a181af9f6ecf5 (DEFAULT_TABLE table=test-workload [id=bc22acb95a754de5892ceea47dbddfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:13.781351 2117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd7fdbeb794b92865a181af9f6ecf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:13.782723 2192 tablet_service.cc:1505] Processing CreateTablet for tablet 04dd7fdbeb794b92865a181af9f6ecf5 (DEFAULT_TABLE table=test-workload [id=bc22acb95a754de5892ceea47dbddfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:13.782699 2243 tablet_bootstrap.cc:492] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: Bootstrap starting.
I20251026 03:09:13.782802 2041 tablet_service.cc:1505] Processing CreateTablet for tablet 04dd7fdbeb794b92865a181af9f6ecf5 (DEFAULT_TABLE table=test-workload [id=bc22acb95a754de5892ceea47dbddfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:13.782859 2192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd7fdbeb794b92865a181af9f6ecf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:13.782902 2041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04dd7fdbeb794b92865a181af9f6ecf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:13.783224 2243 tablet_bootstrap.cc:654] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:13.783897 2243 tablet_bootstrap.cc:492] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: No bootstrap required, opened a new log
I20251026 03:09:13.783941 2243 ts_tablet_manager.cc:1403] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:13.783958 2245 tablet_bootstrap.cc:492] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c: Bootstrap starting.
I20251026 03:09:13.784097 2243 raft_consensus.cc:359] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.784188 2243 raft_consensus.cc:385] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:13.784226 2243 raft_consensus.cc:740] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06765e025bc9477d8f98b95bb9f69a6b, State: Initialized, Role: FOLLOWER
I20251026 03:09:13.784299 2243 consensus_queue.cc:260] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.784372 2245 tablet_bootstrap.cc:654] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:13.784344 2246 tablet_bootstrap.cc:492] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: Bootstrap starting.
I20251026 03:09:13.784488 2243 ts_tablet_manager.cc:1434] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:13.784511 2153 heartbeater.cc:499] Master 127.30.128.126:44977 was elected leader, sending a full tablet report...
I20251026 03:09:13.784888 2246 tablet_bootstrap.cc:654] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:13.784988 2245 tablet_bootstrap.cc:492] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c: No bootstrap required, opened a new log
I20251026 03:09:13.785029 2245 ts_tablet_manager.cc:1403] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:13.785230 2245 raft_consensus.cc:359] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.785305 2245 raft_consensus.cc:385] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:13.785328 2245 raft_consensus.cc:740] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b631b6f2ebc043cf9a2b197c30664e3c, State: Initialized, Role: FOLLOWER
I20251026 03:09:13.785398 2245 consensus_queue.cc:260] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.785442 2246 tablet_bootstrap.cc:492] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: No bootstrap required, opened a new log
I20251026 03:09:13.785549 2245 ts_tablet_manager.cc:1434] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.785579 2228 heartbeater.cc:499] Master 127.30.128.126:44977 was elected leader, sending a full tablet report...
I20251026 03:09:13.786397 2246 ts_tablet_manager.cc:1403] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:09:13.786621 2246 raft_consensus.cc:359] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.786688 2246 raft_consensus.cc:385] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:13.786731 2246 raft_consensus.cc:740] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6b9bc02bab4fb7ac5d8dea9b152e93, State: Initialized, Role: FOLLOWER
I20251026 03:09:13.786774 2246 consensus_queue.cc:260] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.786921 2246 ts_tablet_manager.cc:1434] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:13.786957 2077 heartbeater.cc:499] Master 127.30.128.126:44977 was elected leader, sending a full tablet report...
I20251026 03:09:13.813717 2247 raft_consensus.cc:493] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:09:13.813881 2247 raft_consensus.cc:515] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.814162 2247 leader_election.cc:290] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e6b9bc02bab4fb7ac5d8dea9b152e93 (127.30.128.65:39611), b631b6f2ebc043cf9a2b197c30664e3c (127.30.128.67:33521)
I20251026 03:09:13.817413 2051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5" candidate_uuid: "06765e025bc9477d8f98b95bb9f69a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" is_pre_election: true
I20251026 03:09:13.817510 2051 raft_consensus.cc:2468] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06765e025bc9477d8f98b95bb9f69a6b in term 0.
I20251026 03:09:13.817423 2202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5" candidate_uuid: "06765e025bc9477d8f98b95bb9f69a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" is_pre_election: true
I20251026 03:09:13.817549 2202 raft_consensus.cc:2468] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06765e025bc9477d8f98b95bb9f69a6b in term 0.
I20251026 03:09:13.817673 2093 leader_election.cc:304] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06765e025bc9477d8f98b95bb9f69a6b, 2e6b9bc02bab4fb7ac5d8dea9b152e93; no voters:
I20251026 03:09:13.817761 2247 raft_consensus.cc:2804] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:09:13.817829 2247 raft_consensus.cc:493] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:09:13.817852 2247 raft_consensus.cc:3060] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:13.818425 2247 raft_consensus.cc:515] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.818535 2247 leader_election.cc:290] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [CANDIDATE]: Term 1 election: Requested vote from peers 2e6b9bc02bab4fb7ac5d8dea9b152e93 (127.30.128.65:39611), b631b6f2ebc043cf9a2b197c30664e3c (127.30.128.67:33521)
I20251026 03:09:13.818673 2051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5" candidate_uuid: "06765e025bc9477d8f98b95bb9f69a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93"
I20251026 03:09:13.818720 2051 raft_consensus.cc:3060] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:13.818802 2202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5" candidate_uuid: "06765e025bc9477d8f98b95bb9f69a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b631b6f2ebc043cf9a2b197c30664e3c"
I20251026 03:09:13.818863 2202 raft_consensus.cc:3060] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:13.819339 2051 raft_consensus.cc:2468] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06765e025bc9477d8f98b95bb9f69a6b in term 1.
I20251026 03:09:13.819453 2202 raft_consensus.cc:2468] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06765e025bc9477d8f98b95bb9f69a6b in term 1.
I20251026 03:09:13.819509 2093 leader_election.cc:304] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06765e025bc9477d8f98b95bb9f69a6b, 2e6b9bc02bab4fb7ac5d8dea9b152e93; no voters:
I20251026 03:09:13.819614 2247 raft_consensus.cc:2804] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:13.819741 2247 raft_consensus.cc:697] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 LEADER]: Becoming Leader. State: Replica: 06765e025bc9477d8f98b95bb9f69a6b, State: Running, Role: LEADER
I20251026 03:09:13.819784 2247 consensus_queue.cc:237] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:13.820333 1948 catalog_manager.cc:5649] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b reported cstate change: term changed from 0 to 1, leader changed from <none> to 06765e025bc9477d8f98b95bb9f69a6b (127.30.128.66). New cstate: current_term: 1 leader_uuid: "06765e025bc9477d8f98b95bb9f69a6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } health_report { overall_health: UNKNOWN } } }
I20251026 03:09:13.834478 2202 raft_consensus.cc:1275] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 1 FOLLOWER]: Refusing update from remote peer 06765e025bc9477d8f98b95bb9f69a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:13.834478 2051 raft_consensus.cc:1275] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 1 FOLLOWER]: Refusing update from remote peer 06765e025bc9477d8f98b95bb9f69a6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:13.834729 2253 consensus_queue.cc:1048] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:13.834818 2247 consensus_queue.cc:1048] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:24.287882 2253 raft_consensus.cc:993] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: : Instructing follower 2e6b9bc02bab4fb7ac5d8dea9b152e93 to start an election
I20251026 03:09:24.287986 2253 raft_consensus.cc:1081] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 LEADER]: Signalling peer 2e6b9bc02bab4fb7ac5d8dea9b152e93 to start an election
I20251026 03:09:24.288187 2051 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5"
dest_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93"
from {username='slave'} at 127.0.0.1:46896
I20251026 03:09:24.288254 2051 raft_consensus.cc:493] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:09:24.288275 2051 raft_consensus.cc:3060] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:24.288852 2051 raft_consensus.cc:515] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:24.289037 2051 leader_election.cc:290] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [CANDIDATE]: Term 2 election: Requested vote from peers 06765e025bc9477d8f98b95bb9f69a6b (127.30.128.66:40787), b631b6f2ebc043cf9a2b197c30664e3c (127.30.128.67:33521)
I20251026 03:09:24.291772 2201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5" candidate_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" candidate_term: 2 candidate_status { last_received { term: 1 index: 14875 } } ignore_live_leader: true dest_uuid: "b631b6f2ebc043cf9a2b197c30664e3c"
I20251026 03:09:24.291833 2127 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "04dd7fdbeb794b92865a181af9f6ecf5" candidate_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" candidate_term: 2 candidate_status { last_received { term: 1 index: 14875 } } ignore_live_leader: true dest_uuid: "06765e025bc9477d8f98b95bb9f69a6b"
I20251026 03:09:24.291882 2201 raft_consensus.cc:3060] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:24.291936 2127 raft_consensus.cc:3055] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:09:24.291989 2127 raft_consensus.cc:740] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 06765e025bc9477d8f98b95bb9f69a6b, State: Running, Role: LEADER
I20251026 03:09:24.292054 2127 consensus_queue.cc:260] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14875, Committed index: 14875, Last appended: 1.14875, Last appended by leader: 14875, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:24.292133 2127 raft_consensus.cc:3060] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:24.292620 2201 raft_consensus.cc:2468] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6b9bc02bab4fb7ac5d8dea9b152e93 in term 2.
I20251026 03:09:24.292747 2127 raft_consensus.cc:2468] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e6b9bc02bab4fb7ac5d8dea9b152e93 in term 2.
I20251026 03:09:24.292805 2016 leader_election.cc:304] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e6b9bc02bab4fb7ac5d8dea9b152e93, b631b6f2ebc043cf9a2b197c30664e3c; no voters:
I20251026 03:09:24.293010 2303 raft_consensus.cc:2804] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:09:24.293114 2303 raft_consensus.cc:697] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 2 LEADER]: Becoming Leader. State: Replica: 2e6b9bc02bab4fb7ac5d8dea9b152e93, State: Running, Role: LEADER
I20251026 03:09:24.293190 2303 consensus_queue.cc:237] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14875, Committed index: 14875, Last appended: 1.14875, Last appended by leader: 14875, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } }
I20251026 03:09:24.293627 1946 catalog_manager.cc:5649] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 reported cstate change: term changed from 1 to 2, leader changed from 06765e025bc9477d8f98b95bb9f69a6b (127.30.128.66) to 2e6b9bc02bab4fb7ac5d8dea9b152e93 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06765e025bc9477d8f98b95bb9f69a6b" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e6b9bc02bab4fb7ac5d8dea9b152e93" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b631b6f2ebc043cf9a2b197c30664e3c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 33521 } health_report { overall_health: UNKNOWN } } }
W20251026 03:09:24.297870 2117 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50366: Illegal state: replica 06765e025bc9477d8f98b95bb9f69a6b is not leader of this config: current role FOLLOWER
W20251026 03:09:24.298223 2117 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50366: Illegal state: replica 06765e025bc9477d8f98b95bb9f69a6b is not leader of this config: current role FOLLOWER
W20251026 03:09:24.298512 2117 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50366: Illegal state: replica 06765e025bc9477d8f98b95bb9f69a6b is not leader of this config: current role FOLLOWER
I20251026 03:09:24.298679 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:09:24.302814 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:24.302920 31233 tablet_replica.cc:333] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: stopping tablet replica
I20251026 03:09:24.302995 31233 raft_consensus.cc:2243] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:09:24.303056 31233 pending_rounds.cc:70] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: Trying to abort 1 pending ops.
I20251026 03:09:24.303113 31233 pending_rounds.cc:77] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93: Aborting op as it isn't in flight: id { term: 2 index: 14876 } timestamp: 7214891680945352704 op_type: NO_OP noop_request { }
I20251026 03:09:24.303182 31233 raft_consensus.cc:2889] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:09:24.303217 31233 raft_consensus.cc:2272] T 04dd7fdbeb794b92865a181af9f6ecf5 P 2e6b9bc02bab4fb7ac5d8dea9b152e93 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:24.316890 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:09:24.317720 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:09:24.320055 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:24.320160 31233 tablet_replica.cc:333] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b: stopping tablet replica
I20251026 03:09:24.320206 31233 raft_consensus.cc:2243] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:24.320250 31233 raft_consensus.cc:2272] T 04dd7fdbeb794b92865a181af9f6ecf5 P 06765e025bc9477d8f98b95bb9f69a6b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:24.331660 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:09:24.332439 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:09:24.334808 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:24.334908 31233 tablet_replica.cc:333] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c: stopping tablet replica
I20251026 03:09:24.334960 31233 raft_consensus.cc:2243] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:24.335013 31233 raft_consensus.cc:2272] T 04dd7fdbeb794b92865a181af9f6ecf5 P b631b6f2ebc043cf9a2b197c30664e3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:24.348078 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:09:24.348805 31233 master.cc:561] Master@127.30.128.126:44977 shutting down...
I20251026 03:09:24.350940 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:24.351014 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:24.351055 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P cfd5eb48f88c4a30af1fc1ebc888f232: stopping tablet replica
I20251026 03:09:24.362653 31233 master.cc:583] Master@127.30.128.126:44977 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyNonLeader (10722 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletBootstrapping
I20251026 03:09:24.381562 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:45277
I20251026 03:09:24.381714 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:24.383337 2309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:24.383443 2312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:24.383339 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:24.383402 2310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:24.383683 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:24.383718 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:24.383736 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448164383736 us; error 0 us; skew 500 ppm
I20251026 03:09:24.384277 31233 webserver.cc:492] Webserver started at http://127.30.128.126:34331/ using document root <none> and password file <none>
I20251026 03:09:24.384361 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:24.384410 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:24.384464 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:24.384727 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.385535 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:24.385963 2317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.386090 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.386143 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.386188 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:24.398543 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:24.398742 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:24.401116 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:45277
I20251026 03:09:24.413409 2369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:45277 every 8 connection(s)
I20251026 03:09:24.413538 2370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:24.414561 2370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9: Bootstrap starting.
I20251026 03:09:24.414827 2370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:24.415318 2370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9: No bootstrap required, opened a new log
I20251026 03:09:24.415455 2370 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" member_type: VOTER }
I20251026 03:09:24.415501 2370 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:24.415524 2370 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eb574b7045b4a0e98fd0ca18c58bfd9, State: Initialized, Role: FOLLOWER
I20251026 03:09:24.415573 2370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" member_type: VOTER }
I20251026 03:09:24.415602 2370 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:09:24.415624 2370 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:09:24.415649 2370 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:24.416096 2370 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" member_type: VOTER }
I20251026 03:09:24.416147 2370 leader_election.cc:304] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eb574b7045b4a0e98fd0ca18c58bfd9; no voters:
I20251026 03:09:24.416249 2370 leader_election.cc:290] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:09:24.416307 2373 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:24.416396 2370 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:09:24.416400 2373 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 1 LEADER]: Becoming Leader. State: Replica: 5eb574b7045b4a0e98fd0ca18c58bfd9, State: Running, Role: LEADER
I20251026 03:09:24.416556 2373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" member_type: VOTER }
I20251026 03:09:24.416803 2375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eb574b7045b4a0e98fd0ca18c58bfd9. Latest consensus state: current_term: 1 leader_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" member_type: VOTER } }
I20251026 03:09:24.416814 2374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eb574b7045b4a0e98fd0ca18c58bfd9" member_type: VOTER } }
I20251026 03:09:24.417075 2374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:24.417196 2375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:24.417222 2383 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:09:24.417351 2383 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:09:24.417718 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:09:24.417903 2383 catalog_manager.cc:1357] Generated new cluster ID: 829de51503fb474d8bef5a5110ababcd
I20251026 03:09:24.417948 2383 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:09:24.429912 2383 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:09:24.430303 2383 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:09:24.437793 2383 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9: Generated new TSK 0
I20251026 03:09:24.437877 2383 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:09:24.449720 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:24.451198 2394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:24.451233 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:24.451274 2392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:24.451318 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:24.451510 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:24.451541 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:24.451568 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448164451568 us; error 0 us; skew 500 ppm
I20251026 03:09:24.452154 31233 webserver.cc:492] Webserver started at http://127.30.128.65:42207/ using document root <none> and password file <none>
I20251026 03:09:24.452244 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:24.452298 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:24.452346 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:24.452600 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "9c3b1145d18e40549d4f9b9b4afcec84"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.453431 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:24.453905 2399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.454015 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.454061 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "9c3b1145d18e40549d4f9b9b4afcec84"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.454102 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:24.467599 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:24.467758 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:24.467875 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:24.468019 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:24.468040 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.468060 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:24.468094 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.470649 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:39623
I20251026 03:09:24.470727 2462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:39623 every 8 connection(s)
I20251026 03:09:24.471026 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:24.472134 2467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:24.472294 2470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:24.472365 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:24.472419 2468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:24.472613 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:24.472642 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:24.472664 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448164472664 us; error 0 us; skew 500 ppm
I20251026 03:09:24.473402 31233 webserver.cc:492] Webserver started at http://127.30.128.66:34345/ using document root <none> and password file <none>
I20251026 03:09:24.473484 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:24.473523 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:24.473567 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:24.473878 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "3cb46b327359412a8d24362d220863d3"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.474650 2463 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45277
I20251026 03:09:24.474707 2463 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:24.474802 2463 heartbeater.cc:507] Master 127.30.128.126:45277 requested a full tablet report, sending...
I20251026 03:09:24.474828 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:24.475100 2334 ts_manager.cc:194] Registered new tserver with Master: 9c3b1145d18e40549d4f9b9b4afcec84 (127.30.128.65:39623)
I20251026 03:09:24.475270 2475 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.475400 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.475453 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "3cb46b327359412a8d24362d220863d3"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.475502 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:24.475785 2334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58618
I20251026 03:09:24.482429 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:24.482581 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:24.482717 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:24.482892 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:24.482919 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.482950 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:24.482977 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.485550 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:42395
I20251026 03:09:24.485611 2538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:42395 every 8 connection(s)
I20251026 03:09:24.485903 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:24.486896 2542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:24.487078 2545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:24.486961 2543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:24.487483 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:24.487589 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:24.487613 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:24.487624 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448164487625 us; error 0 us; skew 500 ppm
I20251026 03:09:24.488139 31233 webserver.cc:492] Webserver started at http://127.30.128.67:38533/ using document root <none> and password file <none>
I20251026 03:09:24.488224 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:24.488278 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:24.488325 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:24.488622 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "6f0e829696624718bdf1ce414dee6c8c"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.488880 2539 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45277
I20251026 03:09:24.488931 2539 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:24.489023 2539 heartbeater.cc:507] Master 127.30.128.126:45277 requested a full tablet report, sending...
I20251026 03:09:24.489272 2334 ts_manager.cc:194] Registered new tserver with Master: 3cb46b327359412a8d24362d220863d3 (127.30.128.66:42395)
I20251026 03:09:24.489578 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:24.489681 2334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58632
I20251026 03:09:24.490083 2550 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.490196 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.490250 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "6f0e829696624718bdf1ce414dee6c8c"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:24.490298 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:24.500644 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:24.500810 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:24.500937 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:24.501075 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:24.501098 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.501125 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:24.501142 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.504359 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:44787
I20251026 03:09:24.504395 2613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:44787 every 8 connection(s)
I20251026 03:09:24.507232 2614 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45277
I20251026 03:09:24.507308 2614 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:24.507401 2614 heartbeater.cc:507] Master 127.30.128.126:45277 requested a full tablet report, sending...
I20251026 03:09:24.507640 2334 ts_manager.cc:194] Registered new tserver with Master: 6f0e829696624718bdf1ce414dee6c8c (127.30.128.67:44787)
I20251026 03:09:24.507843 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003327496s
I20251026 03:09:24.508088 2334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58648
I20251026 03:09:24.508229 31233 test_util.cc:276] Using random seed: 472040005
I20251026 03:09:24.512629 2334 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58658:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:09:24.512748 2334 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:09:24.517681 2503 tablet_service.cc:1505] Processing CreateTablet for tablet 939b8b9fa89e47f0b9236bab16c07234 (DEFAULT_TABLE table=test-workload [id=bafa6b8ce44242ac8ca75c875abfd982]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:24.517848 2503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939b8b9fa89e47f0b9236bab16c07234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:24.517915 2578 tablet_service.cc:1505] Processing CreateTablet for tablet 939b8b9fa89e47f0b9236bab16c07234 (DEFAULT_TABLE table=test-workload [id=bafa6b8ce44242ac8ca75c875abfd982]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:24.518025 2578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939b8b9fa89e47f0b9236bab16c07234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:24.518970 2630 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: Bootstrap starting.
I20251026 03:09:24.519202 2427 tablet_service.cc:1505] Processing CreateTablet for tablet 939b8b9fa89e47f0b9236bab16c07234 (DEFAULT_TABLE table=test-workload [id=bafa6b8ce44242ac8ca75c875abfd982]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:24.519215 2631 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: Bootstrap starting.
I20251026 03:09:24.519317 2427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 939b8b9fa89e47f0b9236bab16c07234. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:24.519459 2630 tablet_bootstrap.cc:654] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:24.519632 2631 tablet_bootstrap.cc:654] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:24.520074 2630 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: No bootstrap required, opened a new log
I20251026 03:09:24.520105 2630 ts_tablet_manager.cc:1403] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:24.520176 2631 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: No bootstrap required, opened a new log
I20251026 03:09:24.520227 2631 ts_tablet_manager.cc:1403] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:24.520210 2630 raft_consensus.cc:359] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.520255 2630 raft_consensus.cc:385] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:24.520269 2630 raft_consensus.cc:740] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb46b327359412a8d24362d220863d3, State: Initialized, Role: FOLLOWER
I20251026 03:09:24.520310 2630 consensus_queue.cc:260] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.520370 2631 raft_consensus.cc:359] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.520429 2631 raft_consensus.cc:385] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:24.520448 2631 raft_consensus.cc:740] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f0e829696624718bdf1ce414dee6c8c, State: Initialized, Role: FOLLOWER
I20251026 03:09:24.520496 2539 heartbeater.cc:499] Master 127.30.128.126:45277 was elected leader, sending a full tablet report...
I20251026 03:09:24.520527 2631 consensus_queue.cc:260] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.520452 2630 ts_tablet_manager.cc:1434] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:24.520660 2631 ts_tablet_manager.cc:1434] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:09:24.520702 2614 heartbeater.cc:499] Master 127.30.128.126:45277 was elected leader, sending a full tablet report...
I20251026 03:09:24.521020 2635 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Bootstrap starting.
I20251026 03:09:24.521425 2635 tablet_bootstrap.cc:654] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:24.521935 2635 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: No bootstrap required, opened a new log
I20251026 03:09:24.521988 2635 ts_tablet_manager.cc:1403] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:24.522085 2635 raft_consensus.cc:359] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.522130 2635 raft_consensus.cc:385] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:24.522151 2635 raft_consensus.cc:740] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3b1145d18e40549d4f9b9b4afcec84, State: Initialized, Role: FOLLOWER
I20251026 03:09:24.522236 2635 consensus_queue.cc:260] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.522401 2635 ts_tablet_manager.cc:1434] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:09:24.522435 2463 heartbeater.cc:499] Master 127.30.128.126:45277 was elected leader, sending a full tablet report...
I20251026 03:09:24.650851 2634 raft_consensus.cc:493] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:09:24.650972 2634 raft_consensus.cc:515] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.651167 2634 leader_election.cc:290] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c3b1145d18e40549d4f9b9b4afcec84 (127.30.128.65:39623), 6f0e829696624718bdf1ce414dee6c8c (127.30.128.67:44787)
I20251026 03:09:24.653635 2437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234" candidate_uuid: "3cb46b327359412a8d24362d220863d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" is_pre_election: true
I20251026 03:09:24.653690 2588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234" candidate_uuid: "3cb46b327359412a8d24362d220863d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f0e829696624718bdf1ce414dee6c8c" is_pre_election: true
I20251026 03:09:24.653731 2437 raft_consensus.cc:2468] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb46b327359412a8d24362d220863d3 in term 0.
I20251026 03:09:24.653829 2588 raft_consensus.cc:2468] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb46b327359412a8d24362d220863d3 in term 0.
I20251026 03:09:24.653993 2476 leader_election.cc:304] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb46b327359412a8d24362d220863d3, 9c3b1145d18e40549d4f9b9b4afcec84; no voters:
I20251026 03:09:24.654089 2634 raft_consensus.cc:2804] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:09:24.654127 2634 raft_consensus.cc:493] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:09:24.654206 2634 raft_consensus.cc:3060] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:24.654881 2634 raft_consensus.cc:515] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.655001 2634 leader_election.cc:290] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [CANDIDATE]: Term 1 election: Requested vote from peers 9c3b1145d18e40549d4f9b9b4afcec84 (127.30.128.65:39623), 6f0e829696624718bdf1ce414dee6c8c (127.30.128.67:44787)
I20251026 03:09:24.655141 2588 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234" candidate_uuid: "3cb46b327359412a8d24362d220863d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f0e829696624718bdf1ce414dee6c8c"
I20251026 03:09:24.655151 2437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234" candidate_uuid: "3cb46b327359412a8d24362d220863d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c3b1145d18e40549d4f9b9b4afcec84"
I20251026 03:09:24.655228 2588 raft_consensus.cc:3060] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:24.655236 2437 raft_consensus.cc:3060] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:24.655905 2437 raft_consensus.cc:2468] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb46b327359412a8d24362d220863d3 in term 1.
I20251026 03:09:24.655905 2588 raft_consensus.cc:2468] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb46b327359412a8d24362d220863d3 in term 1.
I20251026 03:09:24.656082 2476 leader_election.cc:304] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb46b327359412a8d24362d220863d3, 9c3b1145d18e40549d4f9b9b4afcec84; no voters:
I20251026 03:09:24.656157 2634 raft_consensus.cc:2804] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:24.656279 2634 raft_consensus.cc:697] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb46b327359412a8d24362d220863d3, State: Running, Role: LEADER
I20251026 03:09:24.656318 2634 consensus_queue.cc:237] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:24.656810 2334 catalog_manager.cc:5649] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cb46b327359412a8d24362d220863d3 (127.30.128.66). New cstate: current_term: 1 leader_uuid: "3cb46b327359412a8d24362d220863d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } health_report { overall_health: HEALTHY } } }
I20251026 03:09:24.668987 2437 raft_consensus.cc:1275] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 1 FOLLOWER]: Refusing update from remote peer 3cb46b327359412a8d24362d220863d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:24.668996 2588 raft_consensus.cc:1275] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 1 FOLLOWER]: Refusing update from remote peer 3cb46b327359412a8d24362d220863d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:24.669281 2634 consensus_queue.cc:1048] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:24.669389 2640 consensus_queue.cc:1048] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:35.078689 2634 raft_consensus.cc:993] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: : Instructing follower 9c3b1145d18e40549d4f9b9b4afcec84 to start an election
I20251026 03:09:35.078773 2634 raft_consensus.cc:1081] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 LEADER]: Signalling peer 9c3b1145d18e40549d4f9b9b4afcec84 to start an election
I20251026 03:09:35.078980 2437 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234"
dest_uuid: "9c3b1145d18e40549d4f9b9b4afcec84"
from {username='slave'} at 127.0.0.1:56802
I20251026 03:09:35.079044 2437 raft_consensus.cc:493] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:09:35.079071 2437 raft_consensus.cc:3060] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:35.079701 2437 raft_consensus.cc:515] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:35.079876 2437 leader_election.cc:290] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [CANDIDATE]: Term 2 election: Requested vote from peers 6f0e829696624718bdf1ce414dee6c8c (127.30.128.67:44787), 3cb46b327359412a8d24362d220863d3 (127.30.128.66:42395)
I20251026 03:09:35.082545 2587 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234" candidate_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" candidate_term: 2 candidate_status { last_received { term: 1 index: 14502 } } ignore_live_leader: true dest_uuid: "6f0e829696624718bdf1ce414dee6c8c"
I20251026 03:09:35.082566 2513 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "939b8b9fa89e47f0b9236bab16c07234" candidate_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" candidate_term: 2 candidate_status { last_received { term: 1 index: 14502 } } ignore_live_leader: true dest_uuid: "3cb46b327359412a8d24362d220863d3"
I20251026 03:09:35.082626 2587 raft_consensus.cc:3060] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:35.082644 2513 raft_consensus.cc:3055] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:09:35.082664 2513 raft_consensus.cc:740] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3cb46b327359412a8d24362d220863d3, State: Running, Role: LEADER
I20251026 03:09:35.082736 2513 consensus_queue.cc:260] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14502, Committed index: 14502, Last appended: 1.14502, Last appended by leader: 14502, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:35.082816 2513 raft_consensus.cc:3060] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:35.083324 2587 raft_consensus.cc:2468] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c3b1145d18e40549d4f9b9b4afcec84 in term 2.
I20251026 03:09:35.083431 2513 raft_consensus.cc:2468] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c3b1145d18e40549d4f9b9b4afcec84 in term 2.
I20251026 03:09:35.083531 2402 leader_election.cc:304] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f0e829696624718bdf1ce414dee6c8c, 9c3b1145d18e40549d4f9b9b4afcec84; no voters:
I20251026 03:09:35.083710 2687 raft_consensus.cc:2804] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:09:35.083832 2687 raft_consensus.cc:697] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 LEADER]: Becoming Leader. State: Replica: 9c3b1145d18e40549d4f9b9b4afcec84, State: Running, Role: LEADER
I20251026 03:09:35.083889 2687 consensus_queue.cc:237] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14502, Committed index: 14502, Last appended: 1.14502, Last appended by leader: 14502, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:35.084342 2334 catalog_manager.cc:5649] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 reported cstate change: term changed from 1 to 2, leader changed from 3cb46b327359412a8d24362d220863d3 (127.30.128.66) to 9c3b1145d18e40549d4f9b9b4afcec84 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } health_report { overall_health: UNKNOWN } } }
I20251026 03:09:35.089952 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:09:35.092183 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:35.092281 31233 tablet_replica.cc:333] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: stopping tablet replica
I20251026 03:09:35.092339 31233 raft_consensus.cc:2243] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:09:35.092389 31233 pending_rounds.cc:70] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Trying to abort 1 pending ops.
I20251026 03:09:35.092407 31233 pending_rounds.cc:77] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Aborting op as it isn't in flight: id { term: 2 index: 14503 } timestamp: 7214891725144162304 op_type: NO_OP noop_request { }
I20251026 03:09:35.092447 31233 raft_consensus.cc:2889] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:09:35.092470 31233 raft_consensus.cc:2272] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:35.106271 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:09:35.107239 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:35.108594 2693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:35.108709 2694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:35.108709 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:35.108604 2696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:35.108903 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:35.108948 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:35.108968 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448175108967 us; error 0 us; skew 500 ppm
I20251026 03:09:35.109570 31233 webserver.cc:492] Webserver started at http://127.30.128.65:42207/ using document root <none> and password file <none>
I20251026 03:09:35.109643 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:35.109678 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:09:35.110225 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:35.110643 2701 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:35.110759 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:35.110817 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "9c3b1145d18e40549d4f9b9b4afcec84"
format_stamp: "Formatted at 2025-10-26 03:09:24 on dist-test-slave-51kf"
I20251026 03:09:35.110873 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletBootstrapping.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:35.136509 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:35.136705 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:35.136854 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:35.137193 2708 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:09:35.137912 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:09:35.137955 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:09:35.137993 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:09:35.138370 31233 ts_tablet_manager.cc:616] Registered 1 tablets
W20251026 03:09:35.138398 2708 ts_tablet_manager.cc:1382] Injecting 1000ms delay in tablet bootstrapping
I20251026 03:09:35.138399 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:35.140985 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:39623
I20251026 03:09:35.147070 2765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:39623 every 8 connection(s)
I20251026 03:09:35.148593 2766 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45277
I20251026 03:09:35.148646 2766 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:35.148756 2766 heartbeater.cc:507] Master 127.30.128.126:45277 requested a full tablet report, sending...
I20251026 03:09:35.149039 2334 ts_manager.cc:194] Re-registered known tserver with Master: 9c3b1145d18e40549d4f9b9b4afcec84 (127.30.128.65:39623)
I20251026 03:09:35.149394 2334 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42272
I20251026 03:09:35.150460 31233 tablet_server.cc:178] TabletServer@127.30.128.65:39623 shutting down...
I20251026 03:09:35.152398 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:36.138789 2708 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Bootstrap starting.
I20251026 03:09:36.831224 2708 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Bootstrap replayed 1/3 log segments. Stats: ops{read=4955 overwritten=0 applied=4952 ignored=0} inserts{seen=247550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20251026 03:09:36.997582 2772 raft_consensus.cc:670] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: failed to trigger leader election: Illegal state: leader elections are disabled
I20251026 03:09:37.584379 2708 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Bootstrap replayed 2/3 log segments. Stats: ops{read=9896 overwritten=0 applied=9893 ignored=0} inserts{seen=494600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:09:38.305047 2708 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Bootstrap replayed 3/3 log segments. Stats: ops{read=14503 overwritten=0 applied=14502 ignored=0} inserts{seen=725050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:09:38.305505 2708 tablet_bootstrap.cc:492] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Bootstrap complete.
I20251026 03:09:38.310029 2708 ts_tablet_manager.cc:1403] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Time spent bootstrapping tablet: real 2.171s user 1.878s sys 0.291s
I20251026 03:09:38.310175 2708 raft_consensus.cc:359] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:38.310248 2708 raft_consensus.cc:740] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3b1145d18e40549d4f9b9b4afcec84, State: Initialized, Role: FOLLOWER
I20251026 03:09:38.310302 2708 consensus_queue.cc:260] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14502, Last appended: 2.14503, Last appended by leader: 14503, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c3b1145d18e40549d4f9b9b4afcec84" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39623 } } peers { permanent_uuid: "6f0e829696624718bdf1ce414dee6c8c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44787 } } peers { permanent_uuid: "3cb46b327359412a8d24362d220863d3" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42395 } }
I20251026 03:09:38.310444 2708 ts_tablet_manager.cc:1434] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:09:38.310576 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:09:38.310647 31233 raft_consensus.cc:2243] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:38.310688 31233 pending_rounds.cc:70] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Trying to abort 1 pending ops.
I20251026 03:09:38.310709 31233 pending_rounds.cc:77] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84: Aborting op as it isn't in flight: id { term: 2 index: 14503 } timestamp: 7214891725144162304 op_type: NO_OP noop_request { }
I20251026 03:09:38.310751 31233 raft_consensus.cc:2889] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:09:38.310781 31233 raft_consensus.cc:2272] T 939b8b9fa89e47f0b9236bab16c07234 P 9c3b1145d18e40549d4f9b9b4afcec84 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:38.324152 31233 tablet_server.cc:195] TabletServer@127.30.128.65:39623 shutdown complete.
I20251026 03:09:38.325150 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:09:38.327607 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:38.327744 31233 tablet_replica.cc:333] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3: stopping tablet replica
I20251026 03:09:38.327809 31233 raft_consensus.cc:2243] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:38.327862 31233 raft_consensus.cc:2272] T 939b8b9fa89e47f0b9236bab16c07234 P 3cb46b327359412a8d24362d220863d3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:38.341261 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:09:38.342331 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:09:38.344904 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:38.345008 31233 tablet_replica.cc:333] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c: stopping tablet replica
I20251026 03:09:38.345057 31233 raft_consensus.cc:2243] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:38.345110 31233 raft_consensus.cc:2272] T 939b8b9fa89e47f0b9236bab16c07234 P 6f0e829696624718bdf1ce414dee6c8c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:38.367074 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:09:38.367861 31233 master.cc:561] Master@127.30.128.126:45277 shutting down...
I20251026 03:09:38.369819 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:38.369889 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:38.369911 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5eb574b7045b4a0e98fd0ca18c58bfd9: stopping tablet replica
I20251026 03:09:38.381450 31233 master.cc:583] Master@127.30.128.126:45277 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletBootstrapping (14018 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletNotRunning
I20251026 03:09:38.400465 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:39179
I20251026 03:09:38.400610 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:38.401762 2777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:38.401892 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:38.401873 2778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:38.401837 2780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:38.402112 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:38.402148 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:38.402175 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448178402174 us; error 0 us; skew 500 ppm
I20251026 03:09:38.402771 31233 webserver.cc:492] Webserver started at http://127.30.128.126:39981/ using document root <none> and password file <none>
I20251026 03:09:38.402855 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:38.402905 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:38.402956 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:38.403216 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "9bc94cf15dab40a7bb32b0343f3db734"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.404080 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:38.404506 2785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.404623 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.404663 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "9bc94cf15dab40a7bb32b0343f3db734"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.404711 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:38.408574 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:38.408720 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:38.410923 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:39179
I20251026 03:09:38.414423 2838 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:38.419306 2837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:39179 every 8 connection(s)
I20251026 03:09:38.419348 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734: Bootstrap starting.
I20251026 03:09:38.419620 2838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:38.420043 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734: No bootstrap required, opened a new log
I20251026 03:09:38.420157 2838 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc94cf15dab40a7bb32b0343f3db734" member_type: VOTER }
I20251026 03:09:38.420204 2838 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:38.420226 2838 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc94cf15dab40a7bb32b0343f3db734, State: Initialized, Role: FOLLOWER
I20251026 03:09:38.420284 2838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc94cf15dab40a7bb32b0343f3db734" member_type: VOTER }
I20251026 03:09:38.420322 2838 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:09:38.420349 2838 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:09:38.420377 2838 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:38.420820 2838 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc94cf15dab40a7bb32b0343f3db734" member_type: VOTER }
I20251026 03:09:38.420871 2838 leader_election.cc:304] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bc94cf15dab40a7bb32b0343f3db734; no voters:
I20251026 03:09:38.420966 2838 leader_election.cc:290] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:09:38.421001 2841 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:38.421092 2838 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:09:38.421118 2841 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc94cf15dab40a7bb32b0343f3db734, State: Running, Role: LEADER
I20251026 03:09:38.421186 2841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc94cf15dab40a7bb32b0343f3db734" member_type: VOTER }
I20251026 03:09:38.421360 2841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bc94cf15dab40a7bb32b0343f3db734. Latest consensus state: current_term: 1 leader_uuid: "9bc94cf15dab40a7bb32b0343f3db734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc94cf15dab40a7bb32b0343f3db734" member_type: VOTER } }
I20251026 03:09:38.421381 2843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bc94cf15dab40a7bb32b0343f3db734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc94cf15dab40a7bb32b0343f3db734" member_type: VOTER } }
I20251026 03:09:38.421444 2841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:38.421584 2843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:38.421733 2848 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:09:38.421895 2848 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:09:38.422410 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:09:38.422513 2848 catalog_manager.cc:1357] Generated new cluster ID: 5f8df895d29344f3a5160a8cdaf679be
I20251026 03:09:38.422564 2848 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:09:38.439791 2848 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:09:38.440219 2848 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:09:38.450409 2848 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734: Generated new TSK 0
I20251026 03:09:38.450486 2848 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:09:38.454339 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:38.455554 2862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:38.455662 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:38.455562 2859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:38.455590 2860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:38.455897 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:38.455932 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:38.455943 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448178455944 us; error 0 us; skew 500 ppm
I20251026 03:09:38.456491 31233 webserver.cc:492] Webserver started at http://127.30.128.65:45387/ using document root <none> and password file <none>
I20251026 03:09:38.456578 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:38.456624 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:38.456671 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:38.456928 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "a4bc7aef5200485c9beaa650d25d759e"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.457683 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:38.458150 2867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.458283 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.458320 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "a4bc7aef5200485c9beaa650d25d759e"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.458359 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:38.464897 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:38.465066 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:38.465199 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:38.465389 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:38.465416 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.465444 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:38.465468 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.467969 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:46829
I20251026 03:09:38.468133 2930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:46829 every 8 connection(s)
I20251026 03:09:38.468343 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:38.469404 2938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:38.469440 2935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:38.469452 2936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:38.469763 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:38.469899 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:38.469928 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:38.469947 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448178469948 us; error 0 us; skew 500 ppm
I20251026 03:09:38.470507 31233 webserver.cc:492] Webserver started at http://127.30.128.66:37143/ using document root <none> and password file <none>
I20251026 03:09:38.470583 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:38.470623 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:38.470678 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:38.470952 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "e76bc639538c445a81e682ffb5f90f17"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.471822 2931 heartbeater.cc:344] Connected to a master server at 127.30.128.126:39179
I20251026 03:09:38.471879 2931 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:38.471941 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:38.471982 2931 heartbeater.cc:507] Master 127.30.128.126:39179 requested a full tablet report, sending...
I20251026 03:09:38.472205 2802 ts_manager.cc:194] Registered new tserver with Master: a4bc7aef5200485c9beaa650d25d759e (127.30.128.65:46829)
I20251026 03:09:38.472499 2943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.472640 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 03:09:38.472687 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "e76bc639538c445a81e682ffb5f90f17"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.472733 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:38.472925 2802 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49632
I20251026 03:09:38.477982 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:38.478161 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:38.478340 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:38.478555 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:38.478582 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.478600 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:38.478614 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.481020 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:35117
I20251026 03:09:38.481065 3006 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:35117 every 8 connection(s)
I20251026 03:09:38.481329 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:38.482326 3010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:38.482375 3011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:38.482331 3013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:38.483095 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:38.483228 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:38.483265 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:38.483278 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448178483278 us; error 0 us; skew 500 ppm
I20251026 03:09:38.483847 31233 webserver.cc:492] Webserver started at http://127.30.128.67:40829/ using document root <none> and password file <none>
I20251026 03:09:38.483930 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:38.483976 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:38.484035 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:38.484313 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "db49a76c5c2f44c6952b4569ff12ca0a"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.484454 3007 heartbeater.cc:344] Connected to a master server at 127.30.128.126:39179
I20251026 03:09:38.484513 3007 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:38.484594 3007 heartbeater.cc:507] Master 127.30.128.126:39179 requested a full tablet report, sending...
I20251026 03:09:38.484791 2802 ts_manager.cc:194] Registered new tserver with Master: e76bc639538c445a81e682ffb5f90f17 (127.30.128.66:35117)
I20251026 03:09:38.485109 2802 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49646
I20251026 03:09:38.485257 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 03:09:38.485677 3018 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.485808 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.485854 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "db49a76c5c2f44c6952b4569ff12ca0a"
format_stamp: "Formatted at 2025-10-26 03:09:38 on dist-test-slave-51kf"
I20251026 03:09:38.485898 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:38.525081 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:38.525332 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:38.525487 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:38.525673 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:38.525698 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.525731 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:38.525759 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.528761 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:44555
I20251026 03:09:38.528843 3081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:44555 every 8 connection(s)
I20251026 03:09:38.535521 3082 heartbeater.cc:344] Connected to a master server at 127.30.128.126:39179
I20251026 03:09:38.535593 3082 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:38.535698 3082 heartbeater.cc:507] Master 127.30.128.126:39179 requested a full tablet report, sending...
I20251026 03:09:38.535910 2802 ts_manager.cc:194] Registered new tserver with Master: db49a76c5c2f44c6952b4569ff12ca0a (127.30.128.67:44555)
I20251026 03:09:38.536214 2802 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49650
I20251026 03:09:38.536620 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007655977s
I20251026 03:09:38.536903 31233 test_util.cc:276] Using random seed: 486068679
I20251026 03:09:38.540882 2802 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49664:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:09:38.541013 2802 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:09:38.545207 2895 tablet_service.cc:1505] Processing CreateTablet for tablet 83c3ed78fa5149ff9eb41fd30dda2127 (DEFAULT_TABLE table=test-workload [id=5494635cf62347afaf38f18b678f2b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:38.545351 2895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c3ed78fa5149ff9eb41fd30dda2127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:38.545806 2971 tablet_service.cc:1505] Processing CreateTablet for tablet 83c3ed78fa5149ff9eb41fd30dda2127 (DEFAULT_TABLE table=test-workload [id=5494635cf62347afaf38f18b678f2b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:38.545899 2971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c3ed78fa5149ff9eb41fd30dda2127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:38.545998 3046 tablet_service.cc:1505] Processing CreateTablet for tablet 83c3ed78fa5149ff9eb41fd30dda2127 (DEFAULT_TABLE table=test-workload [id=5494635cf62347afaf38f18b678f2b2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:38.546083 3046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83c3ed78fa5149ff9eb41fd30dda2127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:38.546864 3097 tablet_bootstrap.cc:492] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Bootstrap starting.
I20251026 03:09:38.547255 3099 tablet_bootstrap.cc:492] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: Bootstrap starting.
I20251026 03:09:38.547304 3097 tablet_bootstrap.cc:654] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:38.547397 3098 tablet_bootstrap.cc:492] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a: Bootstrap starting.
I20251026 03:09:38.547755 3099 tablet_bootstrap.cc:654] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:38.547809 3098 tablet_bootstrap.cc:654] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:38.547931 3097 tablet_bootstrap.cc:492] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: No bootstrap required, opened a new log
I20251026 03:09:38.547976 3097 ts_tablet_manager.cc:1403] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:38.548125 3097 raft_consensus.cc:359] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.548183 3097 raft_consensus.cc:385] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:38.548197 3097 raft_consensus.cc:740] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bc7aef5200485c9beaa650d25d759e, State: Initialized, Role: FOLLOWER
I20251026 03:09:38.548233 3097 consensus_queue.cc:260] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.548362 3097 ts_tablet_manager.cc:1434] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:38.548393 2931 heartbeater.cc:499] Master 127.30.128.126:39179 was elected leader, sending a full tablet report...
I20251026 03:09:38.548413 3098 tablet_bootstrap.cc:492] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a: No bootstrap required, opened a new log
I20251026 03:09:38.548447 3098 ts_tablet_manager.cc:1403] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:38.548714 3099 tablet_bootstrap.cc:492] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: No bootstrap required, opened a new log
I20251026 03:09:38.548784 3099 ts_tablet_manager.cc:1403] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:09:38.548902 3099 raft_consensus.cc:359] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.548954 3099 raft_consensus.cc:385] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:38.548974 3099 raft_consensus.cc:740] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76bc639538c445a81e682ffb5f90f17, State: Initialized, Role: FOLLOWER
I20251026 03:09:38.548686 3098 raft_consensus.cc:359] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.549023 3098 raft_consensus.cc:385] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:38.549046 3098 raft_consensus.cc:740] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db49a76c5c2f44c6952b4569ff12ca0a, State: Initialized, Role: FOLLOWER
I20251026 03:09:38.549116 3098 consensus_queue.cc:260] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.549279 3098 ts_tablet_manager.cc:1434] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:09:38.549315 3082 heartbeater.cc:499] Master 127.30.128.126:39179 was elected leader, sending a full tablet report...
I20251026 03:09:38.549351 3099 consensus_queue.cc:260] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.549489 3099 ts_tablet_manager.cc:1434] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:09:38.549559 3007 heartbeater.cc:499] Master 127.30.128.126:39179 was elected leader, sending a full tablet report...
I20251026 03:09:38.611387 3105 raft_consensus.cc:493] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:09:38.611495 3105 raft_consensus.cc:515] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.611711 3105 leader_election.cc:290] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4bc7aef5200485c9beaa650d25d759e (127.30.128.65:46829), db49a76c5c2f44c6952b4569ff12ca0a (127.30.128.67:44555)
I20251026 03:09:38.614291 2905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127" candidate_uuid: "e76bc639538c445a81e682ffb5f90f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bc7aef5200485c9beaa650d25d759e" is_pre_election: true
I20251026 03:09:38.614291 3056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127" candidate_uuid: "e76bc639538c445a81e682ffb5f90f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" is_pre_election: true
I20251026 03:09:38.614421 2905 raft_consensus.cc:2468] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76bc639538c445a81e682ffb5f90f17 in term 0.
I20251026 03:09:38.614439 3056 raft_consensus.cc:2468] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76bc639538c445a81e682ffb5f90f17 in term 0.
I20251026 03:09:38.614593 2944 leader_election.cc:304] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc7aef5200485c9beaa650d25d759e, e76bc639538c445a81e682ffb5f90f17; no voters:
I20251026 03:09:38.614684 3105 raft_consensus.cc:2804] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:09:38.614723 3105 raft_consensus.cc:493] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:09:38.614742 3105 raft_consensus.cc:3060] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:38.615300 3105 raft_consensus.cc:515] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.615394 3105 leader_election.cc:290] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [CANDIDATE]: Term 1 election: Requested vote from peers a4bc7aef5200485c9beaa650d25d759e (127.30.128.65:46829), db49a76c5c2f44c6952b4569ff12ca0a (127.30.128.67:44555)
I20251026 03:09:38.615558 2905 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127" candidate_uuid: "e76bc639538c445a81e682ffb5f90f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4bc7aef5200485c9beaa650d25d759e"
I20251026 03:09:38.615598 3056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127" candidate_uuid: "e76bc639538c445a81e682ffb5f90f17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db49a76c5c2f44c6952b4569ff12ca0a"
I20251026 03:09:38.615638 2905 raft_consensus.cc:3060] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:38.615644 3056 raft_consensus.cc:3060] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:38.616258 3056 raft_consensus.cc:2468] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76bc639538c445a81e682ffb5f90f17 in term 1.
I20251026 03:09:38.616299 2905 raft_consensus.cc:2468] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76bc639538c445a81e682ffb5f90f17 in term 1.
I20251026 03:09:38.616427 2944 leader_election.cc:304] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc7aef5200485c9beaa650d25d759e, e76bc639538c445a81e682ffb5f90f17; no voters:
I20251026 03:09:38.616520 3105 raft_consensus.cc:2804] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:38.616667 3105 raft_consensus.cc:697] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 LEADER]: Becoming Leader. State: Replica: e76bc639538c445a81e682ffb5f90f17, State: Running, Role: LEADER
I20251026 03:09:38.616724 3105 consensus_queue.cc:237] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:38.617250 2801 catalog_manager.cc:5649] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 reported cstate change: term changed from 0 to 1, leader changed from <none> to e76bc639538c445a81e682ffb5f90f17 (127.30.128.66). New cstate: current_term: 1 leader_uuid: "e76bc639538c445a81e682ffb5f90f17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } health_report { overall_health: UNKNOWN } } }
I20251026 03:09:38.645452 2905 raft_consensus.cc:1275] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 1 FOLLOWER]: Refusing update from remote peer e76bc639538c445a81e682ffb5f90f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:38.645454 3056 raft_consensus.cc:1275] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 1 FOLLOWER]: Refusing update from remote peer e76bc639538c445a81e682ffb5f90f17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:38.645684 3107 consensus_queue.cc:1048] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:38.645814 3105 consensus_queue.cc:1048] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:38.646939 3114 mvcc.cc:204] Tried to move back new op lower bound from 7214891739731447808 to 7214891739614679040. Current Snapshot: MvccSnapshot[applied={T|T < 7214891739730509824}]
I20251026 03:09:49.079607 3133 raft_consensus.cc:993] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: : Instructing follower a4bc7aef5200485c9beaa650d25d759e to start an election
I20251026 03:09:49.079694 3133 raft_consensus.cc:1081] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 LEADER]: Signalling peer a4bc7aef5200485c9beaa650d25d759e to start an election
I20251026 03:09:49.079849 2904 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127"
dest_uuid: "a4bc7aef5200485c9beaa650d25d759e"
from {username='slave'} at 127.0.0.1:55608
I20251026 03:09:49.079917 2904 raft_consensus.cc:493] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:09:49.079936 2904 raft_consensus.cc:3060] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:49.080523 2904 raft_consensus.cc:515] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:49.080660 2904 leader_election.cc:290] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [CANDIDATE]: Term 2 election: Requested vote from peers e76bc639538c445a81e682ffb5f90f17 (127.30.128.66:35117), db49a76c5c2f44c6952b4569ff12ca0a (127.30.128.67:44555)
I20251026 03:09:49.084291 2981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127" candidate_uuid: "a4bc7aef5200485c9beaa650d25d759e" candidate_term: 2 candidate_status { last_received { term: 1 index: 14892 } } ignore_live_leader: true dest_uuid: "e76bc639538c445a81e682ffb5f90f17"
I20251026 03:09:49.084297 3056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "83c3ed78fa5149ff9eb41fd30dda2127" candidate_uuid: "a4bc7aef5200485c9beaa650d25d759e" candidate_term: 2 candidate_status { last_received { term: 1 index: 14892 } } ignore_live_leader: true dest_uuid: "db49a76c5c2f44c6952b4569ff12ca0a"
I20251026 03:09:49.084395 2981 raft_consensus.cc:3055] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:09:49.084398 3056 raft_consensus.cc:3060] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:49.084442 2981 raft_consensus.cc:740] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e76bc639538c445a81e682ffb5f90f17, State: Running, Role: LEADER
I20251026 03:09:49.084510 2981 consensus_queue.cc:260] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14892, Committed index: 14892, Last appended: 1.14892, Last appended by leader: 14892, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:49.084610 2981 raft_consensus.cc:3060] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:09:49.085186 3056 raft_consensus.cc:2468] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc7aef5200485c9beaa650d25d759e in term 2.
I20251026 03:09:49.085286 2981 raft_consensus.cc:2468] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4bc7aef5200485c9beaa650d25d759e in term 2.
I20251026 03:09:49.085361 2868 leader_election.cc:304] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4bc7aef5200485c9beaa650d25d759e, db49a76c5c2f44c6952b4569ff12ca0a; no voters:
I20251026 03:09:49.085587 3151 raft_consensus.cc:2804] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:09:49.085698 3151 raft_consensus.cc:697] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 2 LEADER]: Becoming Leader. State: Replica: a4bc7aef5200485c9beaa650d25d759e, State: Running, Role: LEADER
I20251026 03:09:49.085759 3151 consensus_queue.cc:237] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14892, Committed index: 14892, Last appended: 1.14892, Last appended by leader: 14892, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } }
I20251026 03:09:49.086216 2798 catalog_manager.cc:5649] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e reported cstate change: term changed from 1 to 2, leader changed from e76bc639538c445a81e682ffb5f90f17 (127.30.128.66) to a4bc7aef5200485c9beaa650d25d759e (127.30.128.65). New cstate: current_term: 2 leader_uuid: "a4bc7aef5200485c9beaa650d25d759e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e76bc639538c445a81e682ffb5f90f17" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4bc7aef5200485c9beaa650d25d759e" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db49a76c5c2f44c6952b4569ff12ca0a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 44555 } health_report { overall_health: UNKNOWN } } }
I20251026 03:09:49.091168 31233 tablet_replica.cc:333] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: stopping tablet replica
I20251026 03:09:49.091214 31233 raft_consensus.cc:2243] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:09:49.091245 31233 pending_rounds.cc:70] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Trying to abort 1 pending ops.
I20251026 03:09:49.091255 31233 pending_rounds.cc:77] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Aborting op as it isn't in flight: id { term: 2 index: 14893 } timestamp: 7214891782495825920 op_type: NO_OP noop_request { }
I20251026 03:09:49.091277 31233 raft_consensus.cc:2889] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:09:49.091290 31233 raft_consensus.cc:2272] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:49.093047 31233 ts_tablet_manager.cc:1916] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 03:09:49.094329 31233 ts_tablet_manager.cc:1929] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.14893
I20251026 03:09:49.094385 31233 log.cc:1199] T 83c3ed78fa5149ff9eb41fd30dda2127 P a4bc7aef5200485c9beaa650d25d759e: Deleting WAL directory at /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotRunning.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/83c3ed78fa5149ff9eb41fd30dda2127
I20251026 03:09:49.102106 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:09:49.104486 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:49.116396 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:09:49.117146 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:09:49.119421 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:49.119524 31233 tablet_replica.cc:333] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17: stopping tablet replica
I20251026 03:09:49.119570 31233 raft_consensus.cc:2243] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:49.119608 31233 raft_consensus.cc:2272] T 83c3ed78fa5149ff9eb41fd30dda2127 P e76bc639538c445a81e682ffb5f90f17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:49.141811 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:09:49.142535 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:09:49.145125 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:49.145211 31233 tablet_replica.cc:333] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a: stopping tablet replica
I20251026 03:09:49.145259 31233 raft_consensus.cc:2243] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:49.145318 31233 raft_consensus.cc:2272] T 83c3ed78fa5149ff9eb41fd30dda2127 P db49a76c5c2f44c6952b4569ff12ca0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:49.158334 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:09:49.159009 31233 master.cc:561] Master@127.30.128.126:39179 shutting down...
I20251026 03:09:49.160928 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:49.160989 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:49.161010 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9bc94cf15dab40a7bb32b0343f3db734: stopping tablet replica
I20251026 03:09:49.172565 31233 master.cc:583] Master@127.30.128.126:39179 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletNotRunning (10785 ms)
[ RUN ] TxnParticipantITest.TestProxyTabletNotFound
I20251026 03:09:49.186128 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:43537
I20251026 03:09:49.186264 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:49.187307 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:49.187326 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:49.187433 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:49.187289 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:49.187656 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:49.187696 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:49.187717 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448189187717 us; error 0 us; skew 500 ppm
I20251026 03:09:49.188189 31233 webserver.cc:492] Webserver started at http://127.30.128.126:40651/ using document root <none> and password file <none>
I20251026 03:09:49.188254 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:49.188298 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:49.188342 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:49.188571 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "3d8af99ef4604d94b2a45a144d4e0bf9"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.189327 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.189695 3166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.189890 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.189929 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "3d8af99ef4604d94b2a45a144d4e0bf9"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.189980 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:49.194522 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:49.194705 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:49.196890 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:43537
I20251026 03:09:49.198263 3219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:49.204380 3219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9: Bootstrap starting.
I20251026 03:09:49.204589 3218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:43537 every 8 connection(s)
I20251026 03:09:49.204780 3219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:49.205642 3219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9: No bootstrap required, opened a new log
I20251026 03:09:49.205760 3219 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" member_type: VOTER }
I20251026 03:09:49.205828 3219 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:49.205845 3219 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8af99ef4604d94b2a45a144d4e0bf9, State: Initialized, Role: FOLLOWER
I20251026 03:09:49.205883 3219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" member_type: VOTER }
I20251026 03:09:49.205914 3219 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:09:49.205930 3219 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:09:49.205955 3219 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:49.206359 3219 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" member_type: VOTER }
I20251026 03:09:49.206404 3219 leader_election.cc:304] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8af99ef4604d94b2a45a144d4e0bf9; no voters:
I20251026 03:09:49.206502 3219 leader_election.cc:290] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:09:49.206552 3222 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:49.206633 3219 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:09:49.206694 3222 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d8af99ef4604d94b2a45a144d4e0bf9, State: Running, Role: LEADER
I20251026 03:09:49.206737 3222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" member_type: VOTER }
I20251026 03:09:49.206899 3223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" member_type: VOTER } }
I20251026 03:09:49.206900 3224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8af99ef4604d94b2a45a144d4e0bf9. Latest consensus state: current_term: 1 leader_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8af99ef4604d94b2a45a144d4e0bf9" member_type: VOTER } }
I20251026 03:09:49.206971 3223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:49.206977 3224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:49.207206 3227 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:09:49.207329 3227 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:09:49.207763 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:09:49.208106 3227 catalog_manager.cc:1357] Generated new cluster ID: 79a6a665244248a79187bbea0355914c
I20251026 03:09:49.208140 3227 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:09:49.220345 3227 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:09:49.220715 3227 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:09:49.228981 3227 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9: Generated new TSK 0
I20251026 03:09:49.229043 3227 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:09:49.239419 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:49.240562 3240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:49.240633 3241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:49.240789 3243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:49.240824 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:49.240917 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:49.240949 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:49.240974 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448189240974 us; error 0 us; skew 500 ppm
I20251026 03:09:49.241472 31233 webserver.cc:492] Webserver started at http://127.30.128.65:40659/ using document root <none> and password file <none>
I20251026 03:09:49.241547 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:49.241586 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:49.241634 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:49.241897 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "8dfad104ada24fb28ad0e89a94fa860b"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.242743 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.243145 3248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.243258 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.243297 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "8dfad104ada24fb28ad0e89a94fa860b"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.243345 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:49.254971 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:49.255125 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:49.255249 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:49.255406 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:49.255429 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.255475 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:49.255497 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.258023 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:36871
I20251026 03:09:49.258261 3311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:36871 every 8 connection(s)
I20251026 03:09:49.258541 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:49.259683 3317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:49.259769 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:49.259727 3316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:49.259922 3319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:49.260095 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:49.260126 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:49.260138 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448189260138 us; error 0 us; skew 500 ppm
I20251026 03:09:49.260776 31233 webserver.cc:492] Webserver started at http://127.30.128.66:46535/ using document root <none> and password file <none>
I20251026 03:09:49.260859 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:49.260902 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:49.260948 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:49.261209 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "50c0529b9a414acfbcf33fda42a9cdf1"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.262063 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:49.262112 3312 heartbeater.cc:344] Connected to a master server at 127.30.128.126:43537
I20251026 03:09:49.262154 3312 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:49.262254 3312 heartbeater.cc:507] Master 127.30.128.126:43537 requested a full tablet report, sending...
I20251026 03:09:49.262482 3183 ts_manager.cc:194] Registered new tserver with Master: 8dfad104ada24fb28ad0e89a94fa860b (127.30.128.65:36871)
I20251026 03:09:49.262645 3324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.262820 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.262862 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "50c0529b9a414acfbcf33fda42a9cdf1"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.262908 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:49.263161 3183 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55938
I20251026 03:09:49.283097 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:49.283283 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:49.283432 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:49.283641 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:49.283669 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.283701 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:49.283723 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.286319 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:35445
I20251026 03:09:49.286360 3387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:35445 every 8 connection(s)
I20251026 03:09:49.286651 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:49.287835 3392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:49.287859 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:49.288017 3391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:49.288061 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:49.288225 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:49.288264 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:49.288276 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448189288276 us; error 0 us; skew 500 ppm
I20251026 03:09:49.288885 31233 webserver.cc:492] Webserver started at http://127.30.128.67:46469/ using document root <none> and password file <none>
I20251026 03:09:49.288969 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:49.289013 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:49.289058 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:49.289328 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "e3bd4865988f4c9bb890ebe6c28fa238"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.289408 3388 heartbeater.cc:344] Connected to a master server at 127.30.128.126:43537
I20251026 03:09:49.289470 3388 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:49.289541 3388 heartbeater.cc:507] Master 127.30.128.126:43537 requested a full tablet report, sending...
I20251026 03:09:49.289745 3183 ts_manager.cc:194] Registered new tserver with Master: 50c0529b9a414acfbcf33fda42a9cdf1 (127.30.128.66:35445)
I20251026 03:09:49.290046 3183 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55944
I20251026 03:09:49.290341 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:49.290793 3399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.290900 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.290949 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "e3bd4865988f4c9bb890ebe6c28fa238"
format_stamp: "Formatted at 2025-10-26 03:09:49 on dist-test-slave-51kf"
I20251026 03:09:49.290980 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestProxyTabletNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:49.298555 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:49.298676 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:49.298765 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:49.298887 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:49.298907 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.298923 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:49.298939 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.301606 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:41943
I20251026 03:09:49.301651 3462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:41943 every 8 connection(s)
I20251026 03:09:49.304525 3463 heartbeater.cc:344] Connected to a master server at 127.30.128.126:43537
I20251026 03:09:49.304597 3463 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:49.304700 3463 heartbeater.cc:507] Master 127.30.128.126:43537 requested a full tablet report, sending...
I20251026 03:09:49.304913 3183 ts_manager.cc:194] Registered new tserver with Master: e3bd4865988f4c9bb890ebe6c28fa238 (127.30.128.67:41943)
I20251026 03:09:49.305078 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003291849s
I20251026 03:09:49.305246 3183 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55948
I20251026 03:09:49.305423 31233 test_util.cc:276] Using random seed: 496837198
I20251026 03:09:49.309432 3183 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55956:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:09:49.309551 3183 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:09:49.313668 3352 tablet_service.cc:1505] Processing CreateTablet for tablet 75b6accaa5eb4ba9828714611a2778b6 (DEFAULT_TABLE table=test-workload [id=80452b00c5dc40aea8b32f6c962ca4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:49.313700 3276 tablet_service.cc:1505] Processing CreateTablet for tablet 75b6accaa5eb4ba9828714611a2778b6 (DEFAULT_TABLE table=test-workload [id=80452b00c5dc40aea8b32f6c962ca4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:49.313832 3276 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b6accaa5eb4ba9828714611a2778b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:49.313906 3427 tablet_service.cc:1505] Processing CreateTablet for tablet 75b6accaa5eb4ba9828714611a2778b6 (DEFAULT_TABLE table=test-workload [id=80452b00c5dc40aea8b32f6c962ca4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:49.314000 3427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b6accaa5eb4ba9828714611a2778b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:49.313833 3352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75b6accaa5eb4ba9828714611a2778b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:49.315069 3479 tablet_bootstrap.cc:492] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238: Bootstrap starting.
I20251026 03:09:49.315122 3478 tablet_bootstrap.cc:492] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: Bootstrap starting.
I20251026 03:09:49.315441 3479 tablet_bootstrap.cc:654] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:49.315508 3478 tablet_bootstrap.cc:654] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:49.316030 3478 tablet_bootstrap.cc:492] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: No bootstrap required, opened a new log
I20251026 03:09:49.316035 3479 tablet_bootstrap.cc:492] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238: No bootstrap required, opened a new log
I20251026 03:09:49.316087 3478 ts_tablet_manager.cc:1403] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.316099 3479 ts_tablet_manager.cc:1403] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.316241 3478 raft_consensus.cc:359] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.316243 3479 raft_consensus.cc:359] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.316318 3478 raft_consensus.cc:385] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:49.316471 3479 raft_consensus.cc:385] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:49.316478 3478 raft_consensus.cc:740] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dfad104ada24fb28ad0e89a94fa860b, State: Initialized, Role: FOLLOWER
I20251026 03:09:49.316512 3479 raft_consensus.cc:740] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bd4865988f4c9bb890ebe6c28fa238, State: Initialized, Role: FOLLOWER
I20251026 03:09:49.316552 3478 consensus_queue.cc:260] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.316560 3479 consensus_queue.cc:260] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.316732 3478 ts_tablet_manager.cc:1434] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.316741 3479 ts_tablet_manager.cc:1434] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.316776 3463 heartbeater.cc:499] Master 127.30.128.126:43537 was elected leader, sending a full tablet report...
I20251026 03:09:49.316810 3312 heartbeater.cc:499] Master 127.30.128.126:43537 was elected leader, sending a full tablet report...
I20251026 03:09:49.316817 3482 tablet_bootstrap.cc:492] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1: Bootstrap starting.
I20251026 03:09:49.317286 3482 tablet_bootstrap.cc:654] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:49.317829 3482 tablet_bootstrap.cc:492] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1: No bootstrap required, opened a new log
I20251026 03:09:49.317865 3482 ts_tablet_manager.cc:1403] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:49.317950 3482 raft_consensus.cc:359] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.317984 3482 raft_consensus.cc:385] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:49.317996 3482 raft_consensus.cc:740] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c0529b9a414acfbcf33fda42a9cdf1, State: Initialized, Role: FOLLOWER
I20251026 03:09:49.318032 3482 consensus_queue.cc:260] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.318147 3482 ts_tablet_manager.cc:1434] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:49.318185 3388 heartbeater.cc:499] Master 127.30.128.126:43537 was elected leader, sending a full tablet report...
W20251026 03:09:49.508497 3313 tablet.cc:2378] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:09:49.514984 3483 raft_consensus.cc:493] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:09:49.515086 3483 raft_consensus.cc:515] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.515311 3483 leader_election.cc:290] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50c0529b9a414acfbcf33fda42a9cdf1 (127.30.128.66:35445), e3bd4865988f4c9bb890ebe6c28fa238 (127.30.128.67:41943)
I20251026 03:09:49.518467 3437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b6accaa5eb4ba9828714611a2778b6" candidate_uuid: "8dfad104ada24fb28ad0e89a94fa860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" is_pre_election: true
I20251026 03:09:49.518473 3362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b6accaa5eb4ba9828714611a2778b6" candidate_uuid: "8dfad104ada24fb28ad0e89a94fa860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" is_pre_election: true
I20251026 03:09:49.518553 3437 raft_consensus.cc:2468] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfad104ada24fb28ad0e89a94fa860b in term 0.
I20251026 03:09:49.518574 3362 raft_consensus.cc:2468] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dfad104ada24fb28ad0e89a94fa860b in term 0.
I20251026 03:09:49.518725 3249 leader_election.cc:304] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfad104ada24fb28ad0e89a94fa860b, e3bd4865988f4c9bb890ebe6c28fa238; no voters:
I20251026 03:09:49.518812 3483 raft_consensus.cc:2804] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:09:49.518850 3483 raft_consensus.cc:493] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:09:49.518864 3483 raft_consensus.cc:3060] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:49.519404 3483 raft_consensus.cc:515] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.519502 3483 leader_election.cc:290] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [CANDIDATE]: Term 1 election: Requested vote from peers 50c0529b9a414acfbcf33fda42a9cdf1 (127.30.128.66:35445), e3bd4865988f4c9bb890ebe6c28fa238 (127.30.128.67:41943)
I20251026 03:09:49.519637 3362 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b6accaa5eb4ba9828714611a2778b6" candidate_uuid: "8dfad104ada24fb28ad0e89a94fa860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50c0529b9a414acfbcf33fda42a9cdf1"
I20251026 03:09:49.519641 3437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75b6accaa5eb4ba9828714611a2778b6" candidate_uuid: "8dfad104ada24fb28ad0e89a94fa860b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bd4865988f4c9bb890ebe6c28fa238"
I20251026 03:09:49.519697 3362 raft_consensus.cc:3060] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:49.519713 3437 raft_consensus.cc:3060] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:49.520296 3437 raft_consensus.cc:2468] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfad104ada24fb28ad0e89a94fa860b in term 1.
I20251026 03:09:49.520352 3362 raft_consensus.cc:2468] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dfad104ada24fb28ad0e89a94fa860b in term 1.
I20251026 03:09:49.520459 3249 leader_election.cc:304] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dfad104ada24fb28ad0e89a94fa860b, e3bd4865988f4c9bb890ebe6c28fa238; no voters:
I20251026 03:09:49.520550 3483 raft_consensus.cc:2804] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:49.520679 3483 raft_consensus.cc:697] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 1 LEADER]: Becoming Leader. State: Replica: 8dfad104ada24fb28ad0e89a94fa860b, State: Running, Role: LEADER
I20251026 03:09:49.520732 3483 consensus_queue.cc:237] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } }
I20251026 03:09:49.521229 3182 catalog_manager.cc:5649] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dfad104ada24fb28ad0e89a94fa860b (127.30.128.65). New cstate: current_term: 1 leader_uuid: "8dfad104ada24fb28ad0e89a94fa860b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dfad104ada24fb28ad0e89a94fa860b" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 36871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 } health_report { overall_health: UNKNOWN } } }
I20251026 03:09:49.546048 3437 raft_consensus.cc:1275] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 1 FOLLOWER]: Refusing update from remote peer 8dfad104ada24fb28ad0e89a94fa860b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:49.546090 3362 raft_consensus.cc:1275] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 1 FOLLOWER]: Refusing update from remote peer 8dfad104ada24fb28ad0e89a94fa860b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:49.546290 3488 consensus_queue.cc:1048] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bd4865988f4c9bb890ebe6c28fa238" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:49.546399 3483 consensus_queue.cc:1048] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c0529b9a414acfbcf33fda42a9cdf1" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 35445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:59.550485 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:09:59.552934 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:59.553125 31233 tablet_replica.cc:333] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b: stopping tablet replica
I20251026 03:09:59.553191 31233 raft_consensus.cc:2243] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:59.553258 31233 raft_consensus.cc:2272] T 75b6accaa5eb4ba9828714611a2778b6 P 8dfad104ada24fb28ad0e89a94fa860b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:59.567293 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:09:59.568210 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:09:59.570710 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:59.570824 31233 tablet_replica.cc:333] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1: stopping tablet replica
I20251026 03:09:59.570883 31233 raft_consensus.cc:2243] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:59.570946 31233 raft_consensus.cc:2272] T 75b6accaa5eb4ba9828714611a2778b6 P 50c0529b9a414acfbcf33fda42a9cdf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:59.583784 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:09:59.584599 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:09:59.586982 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:09:59.587078 31233 tablet_replica.cc:333] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238: stopping tablet replica
I20251026 03:09:59.587136 31233 raft_consensus.cc:2243] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:09:59.587196 31233 raft_consensus.cc:2272] T 75b6accaa5eb4ba9828714611a2778b6 P e3bd4865988f4c9bb890ebe6c28fa238 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:59.590396 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:09:59.591089 31233 master.cc:561] Master@127.30.128.126:43537 shutting down...
I20251026 03:09:59.593159 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:09:59.593217 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:09:59.593235 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3d8af99ef4604d94b2a45a144d4e0bf9: stopping tablet replica
I20251026 03:09:59.604820 31233 master.cc:583] Master@127.30.128.126:43537 shutdown complete.
[ OK ] TxnParticipantITest.TestProxyTabletNotFound (10436 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientGetMetadata
I20251026 03:09:59.622893 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:43363
I20251026 03:09:59.623054 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:09:59.624405 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:59.624465 3537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:59.624495 3535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:59.624440 3534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:59.624775 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:59.624811 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:59.624825 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448199624825 us; error 0 us; skew 500 ppm
I20251026 03:09:59.625334 31233 webserver.cc:492] Webserver started at http://127.30.128.126:38931/ using document root <none> and password file <none>
I20251026 03:09:59.625406 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:59.625444 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:59.625478 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:59.625701 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "4e18b0487fff4badbd826fad0f6ce6d0"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.626533 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:59.626915 3542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.627044 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.627094 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "4e18b0487fff4badbd826fad0f6ce6d0"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.627131 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:59.638068 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:59.638226 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:59.640581 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:43363
I20251026 03:09:59.648017 3594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:43363 every 8 connection(s)
I20251026 03:09:59.649884 3595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:59.650884 3595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0: Bootstrap starting.
I20251026 03:09:59.651163 3595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:59.651600 3595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0: No bootstrap required, opened a new log
I20251026 03:09:59.651710 3595 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" member_type: VOTER }
I20251026 03:09:59.651753 3595 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:59.651774 3595 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e18b0487fff4badbd826fad0f6ce6d0, State: Initialized, Role: FOLLOWER
I20251026 03:09:59.651824 3595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" member_type: VOTER }
I20251026 03:09:59.651859 3595 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:09:59.651891 3595 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:09:59.651917 3595 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:59.652319 3595 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" member_type: VOTER }
I20251026 03:09:59.652367 3595 leader_election.cc:304] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e18b0487fff4badbd826fad0f6ce6d0; no voters:
I20251026 03:09:59.652452 3595 leader_election.cc:290] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:09:59.652493 3598 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:59.652578 3595 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:09:59.652654 3598 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 1 LEADER]: Becoming Leader. State: Replica: 4e18b0487fff4badbd826fad0f6ce6d0, State: Running, Role: LEADER
I20251026 03:09:59.652706 3598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" member_type: VOTER }
I20251026 03:09:59.652966 3600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e18b0487fff4badbd826fad0f6ce6d0. Latest consensus state: current_term: 1 leader_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" member_type: VOTER } }
I20251026 03:09:59.652947 3599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e18b0487fff4badbd826fad0f6ce6d0" member_type: VOTER } }
I20251026 03:09:59.653045 3600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:59.653056 3599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [sys.catalog]: This master's current role is: LEADER
I20251026 03:09:59.653205 3605 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:09:59.653345 3605 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:09:59.653813 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:09:59.653877 3605 catalog_manager.cc:1357] Generated new cluster ID: a44d628f29e24bb38a1c3c1e7a2abee5
I20251026 03:09:59.653931 3605 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:09:59.658041 3605 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:09:59.658397 3605 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:09:59.661829 3605 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0: Generated new TSK 0
I20251026 03:09:59.661892 3605 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:09:59.669484 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:59.670727 3616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:59.670797 3619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:59.670732 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:59.670900 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:59.671059 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:59.671101 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:59.671111 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448199671112 us; error 0 us; skew 500 ppm
I20251026 03:09:59.671640 31233 webserver.cc:492] Webserver started at http://127.30.128.65:44707/ using document root <none> and password file <none>
I20251026 03:09:59.671722 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:59.671765 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:59.671809 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:59.672034 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "07010f5e81e7427da55d99f19b01887d"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.672741 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:09:59.673121 3624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.673249 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.673285 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "07010f5e81e7427da55d99f19b01887d"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.673321 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:59.684345 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:59.684479 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:59.684610 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:59.684824 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:59.684851 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.684882 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:59.684911 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.687529 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:34681
I20251026 03:09:59.687573 3687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:34681 every 8 connection(s)
I20251026 03:09:59.687844 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:59.688722 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:59.688737 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:59.688877 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:59.689103 31233 server_base.cc:1047] running on GCE node
I20251026 03:09:59.689209 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:59.689236 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:59.689252 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448199689252 us; error 0 us; skew 500 ppm
I20251026 03:09:59.689926 31233 webserver.cc:492] Webserver started at http://127.30.128.66:38307/ using document root <none> and password file <none>
I20251026 03:09:59.689998 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:59.690042 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:59.690096 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:59.690344 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "222f4b634d9147189c4c2764de1b03d7"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.691061 3688 heartbeater.cc:344] Connected to a master server at 127.30.128.126:43363
I20251026 03:09:59.691116 3688 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:59.691213 3688 heartbeater.cc:507] Master 127.30.128.126:43363 requested a full tablet report, sending...
I20251026 03:09:59.691217 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:59.691448 3559 ts_manager.cc:194] Registered new tserver with Master: 07010f5e81e7427da55d99f19b01887d (127.30.128.65:34681)
I20251026 03:09:59.691650 3700 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.691745 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.691787 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "222f4b634d9147189c4c2764de1b03d7"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.691823 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:59.692145 3559 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46948
I20251026 03:09:59.710392 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:59.710542 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:59.710652 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:59.710789 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:59.710810 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.710829 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:59.710845 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.713467 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:34429
I20251026 03:09:59.713526 3763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:34429 every 8 connection(s)
I20251026 03:09:59.713829 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:09:59.714958 3767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:09:59.714972 3768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:59.715159 31233 server_base.cc:1047] running on GCE node
W20251026 03:09:59.715355 3770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:09:59.715508 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:09:59.715553 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:09:59.715571 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448199715571 us; error 0 us; skew 500 ppm
I20251026 03:09:59.716126 31233 webserver.cc:492] Webserver started at http://127.30.128.67:33697/ using document root <none> and password file <none>
I20251026 03:09:59.716207 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:09:59.716250 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:09:59.716315 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:09:59.716651 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "c2afe8dfdd334325b6433de9480fc2ca"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.716986 3764 heartbeater.cc:344] Connected to a master server at 127.30.128.126:43363
I20251026 03:09:59.717041 3764 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:59.717134 3764 heartbeater.cc:507] Master 127.30.128.126:43363 requested a full tablet report, sending...
I20251026 03:09:59.717320 3559 ts_manager.cc:194] Registered new tserver with Master: 222f4b634d9147189c4c2764de1b03d7 (127.30.128.66:34429)
I20251026 03:09:59.717528 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:59.717799 3559 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46954
I20251026 03:09:59.717998 3775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.718111 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.718152 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "c2afe8dfdd334325b6433de9480fc2ca"
format_stamp: "Formatted at 2025-10-26 03:09:59 on dist-test-slave-51kf"
I20251026 03:09:59.718194 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientGetMetadata.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:09:59.726965 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:09:59.727128 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:09:59.727254 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:09:59.727418 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:09:59.727442 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.727468 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:09:59.727494 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:09:59.730306 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:41115
I20251026 03:09:59.730350 3838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:41115 every 8 connection(s)
I20251026 03:09:59.732590 3839 heartbeater.cc:344] Connected to a master server at 127.30.128.126:43363
I20251026 03:09:59.732636 3839 heartbeater.cc:461] Registering TS with master...
I20251026 03:09:59.732714 3839 heartbeater.cc:507] Master 127.30.128.126:43363 requested a full tablet report, sending...
I20251026 03:09:59.732882 3559 ts_manager.cc:194] Registered new tserver with Master: c2afe8dfdd334325b6433de9480fc2ca (127.30.128.67:41115)
I20251026 03:09:59.733134 3559 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46960
I20251026 03:09:59.733721 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003266106s
I20251026 03:09:59.734028 31233 test_util.cc:276] Using random seed: 507265804
I20251026 03:09:59.737833 3559 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46970:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:09:59.737968 3559 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:09:59.742170 3803 tablet_service.cc:1505] Processing CreateTablet for tablet 360dc44638ac4fa6bff9fa273ab56df1 (DEFAULT_TABLE table=test-workload [id=0070ef94fc5d40c48e04a2d593bc7ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:59.742246 3728 tablet_service.cc:1505] Processing CreateTablet for tablet 360dc44638ac4fa6bff9fa273ab56df1 (DEFAULT_TABLE table=test-workload [id=0070ef94fc5d40c48e04a2d593bc7ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:59.742288 3803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360dc44638ac4fa6bff9fa273ab56df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:59.742326 3652 tablet_service.cc:1505] Processing CreateTablet for tablet 360dc44638ac4fa6bff9fa273ab56df1 (DEFAULT_TABLE table=test-workload [id=0070ef94fc5d40c48e04a2d593bc7ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:09:59.742347 3728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360dc44638ac4fa6bff9fa273ab56df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:59.742393 3652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 360dc44638ac4fa6bff9fa273ab56df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:09:59.743562 3854 tablet_bootstrap.cc:492] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d: Bootstrap starting.
I20251026 03:09:59.743647 3855 tablet_bootstrap.cc:492] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca: Bootstrap starting.
I20251026 03:09:59.744005 3854 tablet_bootstrap.cc:654] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:59.744019 3855 tablet_bootstrap.cc:654] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:59.744201 3856 tablet_bootstrap.cc:492] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7: Bootstrap starting.
I20251026 03:09:59.744630 3856 tablet_bootstrap.cc:654] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7: Neither blocks nor log segments found. Creating new log.
I20251026 03:09:59.744652 3855 tablet_bootstrap.cc:492] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca: No bootstrap required, opened a new log
I20251026 03:09:59.744686 3855 ts_tablet_manager.cc:1403] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:59.744884 3855 raft_consensus.cc:359] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.744964 3855 raft_consensus.cc:385] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:59.744995 3855 raft_consensus.cc:740] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2afe8dfdd334325b6433de9480fc2ca, State: Initialized, Role: FOLLOWER
I20251026 03:09:59.745052 3855 consensus_queue.cc:260] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.745220 3855 ts_tablet_manager.cc:1434] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:09:59.745301 3839 heartbeater.cc:499] Master 127.30.128.126:43363 was elected leader, sending a full tablet report...
I20251026 03:09:59.745314 3856 tablet_bootstrap.cc:492] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7: No bootstrap required, opened a new log
I20251026 03:09:59.745384 3856 ts_tablet_manager.cc:1403] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:09:59.745419 3854 tablet_bootstrap.cc:492] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d: No bootstrap required, opened a new log
I20251026 03:09:59.745481 3854 ts_tablet_manager.cc:1403] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:09:59.745599 3854 raft_consensus.cc:359] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.745666 3854 raft_consensus.cc:385] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:59.745651 3856 raft_consensus.cc:359] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.745736 3856 raft_consensus.cc:385] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:09:59.745765 3856 raft_consensus.cc:740] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222f4b634d9147189c4c2764de1b03d7, State: Initialized, Role: FOLLOWER
I20251026 03:09:59.745688 3854 raft_consensus.cc:740] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07010f5e81e7427da55d99f19b01887d, State: Initialized, Role: FOLLOWER
I20251026 03:09:59.745882 3856 consensus_queue.cc:260] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.745923 3854 consensus_queue.cc:260] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.746060 3764 heartbeater.cc:499] Master 127.30.128.126:43363 was elected leader, sending a full tablet report...
I20251026 03:09:59.746102 3688 heartbeater.cc:499] Master 127.30.128.126:43363 was elected leader, sending a full tablet report...
I20251026 03:09:59.746061 3856 ts_tablet_manager.cc:1434] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:09:59.746088 3854 ts_tablet_manager.cc:1434] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:09:59.931509 3862 raft_consensus.cc:493] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:09:59.931622 3862 raft_consensus.cc:515] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.931793 3862 leader_election.cc:290] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2afe8dfdd334325b6433de9480fc2ca (127.30.128.67:41115), 222f4b634d9147189c4c2764de1b03d7 (127.30.128.66:34429)
I20251026 03:09:59.934283 3813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "360dc44638ac4fa6bff9fa273ab56df1" candidate_uuid: "07010f5e81e7427da55d99f19b01887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2afe8dfdd334325b6433de9480fc2ca" is_pre_election: true
I20251026 03:09:59.934383 3813 raft_consensus.cc:2468] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07010f5e81e7427da55d99f19b01887d in term 0.
I20251026 03:09:59.934418 3738 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "360dc44638ac4fa6bff9fa273ab56df1" candidate_uuid: "07010f5e81e7427da55d99f19b01887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222f4b634d9147189c4c2764de1b03d7" is_pre_election: true
I20251026 03:09:59.934501 3738 raft_consensus.cc:2468] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07010f5e81e7427da55d99f19b01887d in term 0.
I20251026 03:09:59.934542 3625 leader_election.cc:304] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07010f5e81e7427da55d99f19b01887d, c2afe8dfdd334325b6433de9480fc2ca; no voters:
I20251026 03:09:59.934648 3862 raft_consensus.cc:2804] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:09:59.934691 3862 raft_consensus.cc:493] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:09:59.934721 3862 raft_consensus.cc:3060] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:59.935381 3862 raft_consensus.cc:515] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.935504 3862 leader_election.cc:290] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [CANDIDATE]: Term 1 election: Requested vote from peers c2afe8dfdd334325b6433de9480fc2ca (127.30.128.67:41115), 222f4b634d9147189c4c2764de1b03d7 (127.30.128.66:34429)
I20251026 03:09:59.935691 3738 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "360dc44638ac4fa6bff9fa273ab56df1" candidate_uuid: "07010f5e81e7427da55d99f19b01887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222f4b634d9147189c4c2764de1b03d7"
I20251026 03:09:59.935720 3813 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "360dc44638ac4fa6bff9fa273ab56df1" candidate_uuid: "07010f5e81e7427da55d99f19b01887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2afe8dfdd334325b6433de9480fc2ca"
I20251026 03:09:59.935743 3738 raft_consensus.cc:3060] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:59.935777 3813 raft_consensus.cc:3060] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:09:59.936419 3738 raft_consensus.cc:2468] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07010f5e81e7427da55d99f19b01887d in term 1.
I20251026 03:09:59.936446 3813 raft_consensus.cc:2468] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07010f5e81e7427da55d99f19b01887d in term 1.
I20251026 03:09:59.936578 3625 leader_election.cc:304] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07010f5e81e7427da55d99f19b01887d, c2afe8dfdd334325b6433de9480fc2ca; no voters:
I20251026 03:09:59.936663 3862 raft_consensus.cc:2804] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:09:59.936779 3862 raft_consensus.cc:697] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 1 LEADER]: Becoming Leader. State: Replica: 07010f5e81e7427da55d99f19b01887d, State: Running, Role: LEADER
I20251026 03:09:59.936816 3862 consensus_queue.cc:237] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } }
I20251026 03:09:59.937297 3559 catalog_manager.cc:5649] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d reported cstate change: term changed from 0 to 1, leader changed from <none> to 07010f5e81e7427da55d99f19b01887d (127.30.128.65). New cstate: current_term: 1 leader_uuid: "07010f5e81e7427da55d99f19b01887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07010f5e81e7427da55d99f19b01887d" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34681 } health_report { overall_health: HEALTHY } } }
I20251026 03:09:59.973963 3738 raft_consensus.cc:1275] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 1 FOLLOWER]: Refusing update from remote peer 07010f5e81e7427da55d99f19b01887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:59.973963 3813 raft_consensus.cc:1275] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 1 FOLLOWER]: Refusing update from remote peer 07010f5e81e7427da55d99f19b01887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:09:59.974270 3864 consensus_queue.cc:1048] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "222f4b634d9147189c4c2764de1b03d7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:09:59.974354 3862 consensus_queue.cc:1048] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2afe8dfdd334325b6433de9480fc2ca" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:09.984843 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:10:09.987696 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:09.987803 31233 tablet_replica.cc:333] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d: stopping tablet replica
I20251026 03:10:09.987857 31233 raft_consensus.cc:2243] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:09.987923 31233 raft_consensus.cc:2272] T 360dc44638ac4fa6bff9fa273ab56df1 P 07010f5e81e7427da55d99f19b01887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:10.001744 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:10:10.002593 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:10:10.005602 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:10.005719 31233 tablet_replica.cc:333] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7: stopping tablet replica
I20251026 03:10:10.005836 31233 raft_consensus.cc:2243] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:10.005889 31233 raft_consensus.cc:2272] T 360dc44638ac4fa6bff9fa273ab56df1 P 222f4b634d9147189c4c2764de1b03d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:10.027792 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:10:10.028626 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:10:10.031523 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:10.031649 31233 tablet_replica.cc:333] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca: stopping tablet replica
I20251026 03:10:10.031730 31233 raft_consensus.cc:2243] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:10.031770 31233 raft_consensus.cc:2272] T 360dc44638ac4fa6bff9fa273ab56df1 P c2afe8dfdd334325b6433de9480fc2ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:10.035053 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:10:10.035830 31233 master.cc:561] Master@127.30.128.126:43363 shutting down...
I20251026 03:10:10.038130 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:10.038188 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:10.038204 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4e18b0487fff4badbd826fad0f6ce6d0: stopping tablet replica
I20251026 03:10:10.049870 31233 master.cc:583] Master@127.30.128.126:43363 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientGetMetadata (10445 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock
I20251026 03:10:10.068888 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:40143
I20251026 03:10:10.069024 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:10:10.070360 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:10.070410 3919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:10.070473 3917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:10.070644 3916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:10.070772 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:10.070811 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:10.070822 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448210070822 us; error 0 us; skew 500 ppm
I20251026 03:10:10.071303 31233 webserver.cc:492] Webserver started at http://127.30.128.126:45819/ using document root <none> and password file <none>
I20251026 03:10:10.071364 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:10.071394 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:10.071426 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:10.071671 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "6aaf9424ecb34a39830bc01eadad044f"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.072412 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:10.072762 3924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.072883 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.072925 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "6aaf9424ecb34a39830bc01eadad044f"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.072973 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:10.080250 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:10.080382 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:10.082634 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:40143
I20251026 03:10:10.091157 3976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:40143 every 8 connection(s)
I20251026 03:10:10.092236 3977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:10.093163 3977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f: Bootstrap starting.
I20251026 03:10:10.093420 3977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:10.093906 3977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f: No bootstrap required, opened a new log
I20251026 03:10:10.094022 3977 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf9424ecb34a39830bc01eadad044f" member_type: VOTER }
I20251026 03:10:10.094074 3977 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:10.094089 3977 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaf9424ecb34a39830bc01eadad044f, State: Initialized, Role: FOLLOWER
I20251026 03:10:10.094135 3977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf9424ecb34a39830bc01eadad044f" member_type: VOTER }
I20251026 03:10:10.094169 3977 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:10:10.094194 3977 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:10:10.094219 3977 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:10.094676 3977 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf9424ecb34a39830bc01eadad044f" member_type: VOTER }
I20251026 03:10:10.094724 3977 leader_election.cc:304] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aaf9424ecb34a39830bc01eadad044f; no voters:
I20251026 03:10:10.094844 3977 leader_election.cc:290] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:10:10.094908 3980 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:10.094985 3977 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:10:10.095062 3980 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 1 LEADER]: Becoming Leader. State: Replica: 6aaf9424ecb34a39830bc01eadad044f, State: Running, Role: LEADER
I20251026 03:10:10.095116 3980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf9424ecb34a39830bc01eadad044f" member_type: VOTER }
I20251026 03:10:10.095309 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aaf9424ecb34a39830bc01eadad044f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf9424ecb34a39830bc01eadad044f" member_type: VOTER } }
I20251026 03:10:10.095431 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:10.095336 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aaf9424ecb34a39830bc01eadad044f. Latest consensus state: current_term: 1 leader_uuid: "6aaf9424ecb34a39830bc01eadad044f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaf9424ecb34a39830bc01eadad044f" member_type: VOTER } }
I20251026 03:10:10.095599 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:10.095651 3988 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:10:10.095813 3988 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:10:10.096320 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:10:10.096402 3988 catalog_manager.cc:1357] Generated new cluster ID: 66c1b89263c0419099b81f6369b96565
I20251026 03:10:10.096441 3988 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:10:10.115729 3988 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:10:10.116179 3988 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:10:10.131475 3988 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f: Generated new TSK 0
I20251026 03:10:10.131552 3988 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:10:10.160424 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:10.161751 3999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:10.161932 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:10.161754 4001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:10.161746 3998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:10.162127 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:10.162158 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:10.162169 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448210162170 us; error 0 us; skew 500 ppm
I20251026 03:10:10.162714 31233 webserver.cc:492] Webserver started at http://127.30.128.65:36561/ using document root <none> and password file <none>
I20251026 03:10:10.162814 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:10.162856 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:10.162904 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:10.163158 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "76823839efc54c5eb1cd20589e0ca6ed"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.163993 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:10.164376 4006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.164521 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.164566 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "76823839efc54c5eb1cd20589e0ca6ed"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.164600 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:10.168805 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:10.168936 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:10.169036 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:10.169179 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:10.169200 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.169225 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:10.169240 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.171748 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:41319
I20251026 03:10:10.171800 4069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:41319 every 8 connection(s)
I20251026 03:10:10.172115 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:10.173134 4075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:10.173218 4077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:10.173134 4074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:10.173389 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:10.173565 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:10.173615 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:10.173635 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448210173634 us; error 0 us; skew 500 ppm
I20251026 03:10:10.174343 31233 webserver.cc:492] Webserver started at http://127.30.128.66:45743/ using document root <none> and password file <none>
I20251026 03:10:10.174427 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:10.174469 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:10.174513 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:10.174780 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "73c8e61facbe4889a599878198834b89"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.175498 4070 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40143
I20251026 03:10:10.175539 4070 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:10.175626 4070 heartbeater.cc:507] Master 127.30.128.126:40143 requested a full tablet report, sending...
I20251026 03:10:10.175678 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:10.175872 3941 ts_manager.cc:194] Registered new tserver with Master: 76823839efc54c5eb1cd20589e0ca6ed (127.30.128.65:41319)
I20251026 03:10:10.176185 4082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.176308 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.176360 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "73c8e61facbe4889a599878198834b89"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.176409 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:10.176558 3941 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52340
I20251026 03:10:10.183157 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:10.183344 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:10.183472 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:10.183645 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:10.183670 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.183702 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:10.183728 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.186510 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:33343
I20251026 03:10:10.186558 4145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:33343 every 8 connection(s)
I20251026 03:10:10.186869 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:10.187945 4150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:10.188015 4149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:10.188226 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:10.188301 4152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:10.188455 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:10.188501 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:10.188519 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448210188518 us; error 0 us; skew 500 ppm
I20251026 03:10:10.189121 31233 webserver.cc:492] Webserver started at http://127.30.128.67:35979/ using document root <none> and password file <none>
I20251026 03:10:10.189203 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:10.189246 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:10.189299 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:10.189543 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "24ee1f49b0f645409c6e5d8c3c0a7716"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.189884 4146 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40143
I20251026 03:10:10.189934 4146 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:10.190026 4146 heartbeater.cc:507] Master 127.30.128.126:40143 requested a full tablet report, sending...
I20251026 03:10:10.190261 3941 ts_manager.cc:194] Registered new tserver with Master: 73c8e61facbe4889a599878198834b89 (127.30.128.66:33343)
I20251026 03:10:10.190500 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:10.190578 3941 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52354
I20251026 03:10:10.190991 4157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.191108 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.191154 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "24ee1f49b0f645409c6e5d8c3c0a7716"
format_stamp: "Formatted at 2025-10-26 03:10:10 on dist-test-slave-51kf"
I20251026 03:10:10.191197 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:10.195986 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:10.196189 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:10.196358 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:10.196543 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:10.196573 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.196602 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:10.196630 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.199682 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:45499
I20251026 03:10:10.199733 4220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:45499 every 8 connection(s)
I20251026 03:10:10.202165 4221 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40143
I20251026 03:10:10.202214 4221 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:10.202307 4221 heartbeater.cc:507] Master 127.30.128.126:40143 requested a full tablet report, sending...
I20251026 03:10:10.202502 3941 ts_manager.cc:194] Registered new tserver with Master: 24ee1f49b0f645409c6e5d8c3c0a7716 (127.30.128.67:45499)
I20251026 03:10:10.202816 3941 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52362
I20251026 03:10:10.203145 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003290034s
I20251026 03:10:10.203480 31233 test_util.cc:276] Using random seed: 517735255
I20251026 03:10:10.207768 3941 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52378:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:10:10.207903 3941 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:10:10.212033 4185 tablet_service.cc:1505] Processing CreateTablet for tablet c7e8905fadf4459ea1df3911271f81bf (DEFAULT_TABLE table=test-workload [id=ffa6a14ada40497da82c5ffb4872dfcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:10.212175 4185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7e8905fadf4459ea1df3911271f81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:10.212213 4110 tablet_service.cc:1505] Processing CreateTablet for tablet c7e8905fadf4459ea1df3911271f81bf (DEFAULT_TABLE table=test-workload [id=ffa6a14ada40497da82c5ffb4872dfcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:10.212309 4110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7e8905fadf4459ea1df3911271f81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:10.212718 4034 tablet_service.cc:1505] Processing CreateTablet for tablet c7e8905fadf4459ea1df3911271f81bf (DEFAULT_TABLE table=test-workload [id=ffa6a14ada40497da82c5ffb4872dfcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:10.212805 4034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7e8905fadf4459ea1df3911271f81bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:10.213284 4236 tablet_bootstrap.cc:492] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716: Bootstrap starting.
I20251026 03:10:10.213640 4237 tablet_bootstrap.cc:492] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: Bootstrap starting.
I20251026 03:10:10.213832 4236 tablet_bootstrap.cc:654] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:10.214092 4237 tablet_bootstrap.cc:654] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:10.214262 4238 tablet_bootstrap.cc:492] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed: Bootstrap starting.
I20251026 03:10:10.214437 4236 tablet_bootstrap.cc:492] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716: No bootstrap required, opened a new log
I20251026 03:10:10.214479 4236 ts_tablet_manager.cc:1403] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:10.214596 4238 tablet_bootstrap.cc:654] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:10.214612 4236 raft_consensus.cc:359] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.214671 4236 raft_consensus.cc:385] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:10.214696 4236 raft_consensus.cc:740] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ee1f49b0f645409c6e5d8c3c0a7716, State: Initialized, Role: FOLLOWER
I20251026 03:10:10.214744 4236 consensus_queue.cc:260] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.214886 4236 ts_tablet_manager.cc:1434] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:10:10.214946 4221 heartbeater.cc:499] Master 127.30.128.126:40143 was elected leader, sending a full tablet report...
I20251026 03:10:10.215070 4238 tablet_bootstrap.cc:492] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed: No bootstrap required, opened a new log
I20251026 03:10:10.215144 4238 ts_tablet_manager.cc:1403] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:10.215281 4237 tablet_bootstrap.cc:492] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: No bootstrap required, opened a new log
I20251026 03:10:10.215317 4237 ts_tablet_manager.cc:1403] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 03:10:10.215304 4238 raft_consensus.cc:359] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.215355 4238 raft_consensus.cc:385] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:10.215375 4238 raft_consensus.cc:740] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76823839efc54c5eb1cd20589e0ca6ed, State: Initialized, Role: FOLLOWER
I20251026 03:10:10.215442 4237 raft_consensus.cc:359] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.215502 4237 raft_consensus.cc:385] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:10.215525 4237 raft_consensus.cc:740] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8e61facbe4889a599878198834b89, State: Initialized, Role: FOLLOWER
I20251026 03:10:10.215566 4237 consensus_queue.cc:260] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.215691 4237 ts_tablet_manager.cc:1434] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:10.215727 4146 heartbeater.cc:499] Master 127.30.128.126:40143 was elected leader, sending a full tablet report...
I20251026 03:10:10.215436 4238 consensus_queue.cc:260] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.215991 4238 ts_tablet_manager.cc:1434] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:10:10.216033 4070 heartbeater.cc:499] Master 127.30.128.126:40143 was elected leader, sending a full tablet report...
I20251026 03:10:10.304054 4243 raft_consensus.cc:493] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:10:10.304147 4243 raft_consensus.cc:515] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.304351 4243 leader_election.cc:290] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ee1f49b0f645409c6e5d8c3c0a7716 (127.30.128.67:45499), 76823839efc54c5eb1cd20589e0ca6ed (127.30.128.65:41319)
I20251026 03:10:10.306790 4195 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf" candidate_uuid: "73c8e61facbe4889a599878198834b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" is_pre_election: true
I20251026 03:10:10.306893 4195 raft_consensus.cc:2468] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8e61facbe4889a599878198834b89 in term 0.
I20251026 03:10:10.306864 4044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf" candidate_uuid: "73c8e61facbe4889a599878198834b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76823839efc54c5eb1cd20589e0ca6ed" is_pre_election: true
I20251026 03:10:10.306967 4044 raft_consensus.cc:2468] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8e61facbe4889a599878198834b89 in term 0.
I20251026 03:10:10.307087 4086 leader_election.cc:304] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ee1f49b0f645409c6e5d8c3c0a7716, 73c8e61facbe4889a599878198834b89; no voters:
I20251026 03:10:10.307209 4243 raft_consensus.cc:2804] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:10:10.307253 4243 raft_consensus.cc:493] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:10:10.307272 4243 raft_consensus.cc:3060] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:10.307868 4243 raft_consensus.cc:515] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.307982 4243 leader_election.cc:290] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [CANDIDATE]: Term 1 election: Requested vote from peers 24ee1f49b0f645409c6e5d8c3c0a7716 (127.30.128.67:45499), 76823839efc54c5eb1cd20589e0ca6ed (127.30.128.65:41319)
I20251026 03:10:10.308089 4195 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf" candidate_uuid: "73c8e61facbe4889a599878198834b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716"
I20251026 03:10:10.308094 4044 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf" candidate_uuid: "73c8e61facbe4889a599878198834b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76823839efc54c5eb1cd20589e0ca6ed"
I20251026 03:10:10.308147 4195 raft_consensus.cc:3060] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:10.308166 4044 raft_consensus.cc:3060] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:10.308760 4195 raft_consensus.cc:2468] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8e61facbe4889a599878198834b89 in term 1.
I20251026 03:10:10.308825 4044 raft_consensus.cc:2468] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8e61facbe4889a599878198834b89 in term 1.
I20251026 03:10:10.308923 4086 leader_election.cc:304] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ee1f49b0f645409c6e5d8c3c0a7716, 73c8e61facbe4889a599878198834b89; no voters:
I20251026 03:10:10.309005 4243 raft_consensus.cc:2804] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:10.309120 4243 raft_consensus.cc:697] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 LEADER]: Becoming Leader. State: Replica: 73c8e61facbe4889a599878198834b89, State: Running, Role: LEADER
I20251026 03:10:10.309160 4243 consensus_queue.cc:237] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:10.309635 3940 catalog_manager.cc:5649] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c8e61facbe4889a599878198834b89 (127.30.128.66). New cstate: current_term: 1 leader_uuid: "73c8e61facbe4889a599878198834b89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } health_report { overall_health: UNKNOWN } } }
I20251026 03:10:10.336259 4195 raft_consensus.cc:1275] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 1 FOLLOWER]: Refusing update from remote peer 73c8e61facbe4889a599878198834b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:10.336259 4044 raft_consensus.cc:1275] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 1 FOLLOWER]: Refusing update from remote peer 73c8e61facbe4889a599878198834b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:10.336513 4243 consensus_queue.cc:1048] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:10.336597 4246 consensus_queue.cc:1048] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:10.339901 4255 mvcc.cc:204] Tried to move back new op lower bound from 7214891869537394688 to 7214891869426774016. Current Snapshot: MvccSnapshot[applied={T|T < 7214891869535965184}]
I20251026 03:10:20.884884 4292 raft_consensus.cc:993] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: : Instructing follower 76823839efc54c5eb1cd20589e0ca6ed to start an election
I20251026 03:10:20.885007 4292 raft_consensus.cc:1081] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 LEADER]: Signalling peer 76823839efc54c5eb1cd20589e0ca6ed to start an election
I20251026 03:10:20.885198 4044 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf"
dest_uuid: "76823839efc54c5eb1cd20589e0ca6ed"
from {username='slave'} at 127.0.0.1:54232
I20251026 03:10:20.885284 4044 raft_consensus.cc:493] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:10:20.885316 4044 raft_consensus.cc:3060] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:20.885974 4044 raft_consensus.cc:515] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:20.886138 4044 leader_election.cc:290] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [CANDIDATE]: Term 2 election: Requested vote from peers 73c8e61facbe4889a599878198834b89 (127.30.128.66:33343), 24ee1f49b0f645409c6e5d8c3c0a7716 (127.30.128.67:45499)
I20251026 03:10:20.888590 4194 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf" candidate_uuid: "76823839efc54c5eb1cd20589e0ca6ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 15121 } } ignore_live_leader: true dest_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716"
I20251026 03:10:20.888669 4194 raft_consensus.cc:3060] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:20.888680 4120 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7e8905fadf4459ea1df3911271f81bf" candidate_uuid: "76823839efc54c5eb1cd20589e0ca6ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 15121 } } ignore_live_leader: true dest_uuid: "73c8e61facbe4889a599878198834b89"
I20251026 03:10:20.888759 4120 raft_consensus.cc:3055] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:10:20.888794 4120 raft_consensus.cc:740] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73c8e61facbe4889a599878198834b89, State: Running, Role: LEADER
I20251026 03:10:20.888868 4120 consensus_queue.cc:260] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15121, Committed index: 15121, Last appended: 1.15121, Last appended by leader: 15121, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:20.888943 4120 raft_consensus.cc:3060] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:20.889325 4194 raft_consensus.cc:2468] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76823839efc54c5eb1cd20589e0ca6ed in term 2.
I20251026 03:10:20.889496 4010 leader_election.cc:304] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ee1f49b0f645409c6e5d8c3c0a7716, 76823839efc54c5eb1cd20589e0ca6ed; no voters:
I20251026 03:10:20.889503 4120 raft_consensus.cc:2468] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76823839efc54c5eb1cd20589e0ca6ed in term 2.
I20251026 03:10:20.889732 4298 raft_consensus.cc:2804] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:10:20.889863 4298 raft_consensus.cc:697] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 2 LEADER]: Becoming Leader. State: Replica: 76823839efc54c5eb1cd20589e0ca6ed, State: Running, Role: LEADER
I20251026 03:10:20.889923 4298 consensus_queue.cc:237] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15121, Committed index: 15121, Last appended: 1.15121, Last appended by leader: 15121, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } }
I20251026 03:10:20.890321 3937 catalog_manager.cc:5649] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed reported cstate change: term changed from 1 to 2, leader changed from 73c8e61facbe4889a599878198834b89 (127.30.128.66) to 76823839efc54c5eb1cd20589e0ca6ed (127.30.128.65). New cstate: current_term: 2 leader_uuid: "76823839efc54c5eb1cd20589e0ca6ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76823839efc54c5eb1cd20589e0ca6ed" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 41319 } health_report { overall_health: HEALTHY } } }
I20251026 03:10:20.899868 4194 raft_consensus.cc:1275] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 2 FOLLOWER]: Refusing update from remote peer 76823839efc54c5eb1cd20589e0ca6ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 15121. Preceding OpId from leader: term: 2 index: 15123. (index mismatch)
I20251026 03:10:20.899889 4120 raft_consensus.cc:1275] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 2 FOLLOWER]: Refusing update from remote peer 76823839efc54c5eb1cd20589e0ca6ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 15121. Preceding OpId from leader: term: 2 index: 15123. (index mismatch)
I20251026 03:10:20.900012 4298 consensus_queue.cc:1048] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ee1f49b0f645409c6e5d8c3c0a7716" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15122, Last known committed idx: 15121, Time since last communication: 0.000s
I20251026 03:10:20.900085 4299 consensus_queue.cc:1048] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8e61facbe4889a599878198834b89" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15122, Last known committed idx: 15121, Time since last communication: 0.000s
I20251026 03:10:20.904789 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:10:20.906850 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:20.906975 31233 tablet_replica.cc:333] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed: stopping tablet replica
I20251026 03:10:20.907033 31233 raft_consensus.cc:2243] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:10:20.907092 31233 raft_consensus.cc:2272] T c7e8905fadf4459ea1df3911271f81bf P 76823839efc54c5eb1cd20589e0ca6ed [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:20.920599 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:10:20.921288 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:10:20.923691 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:20.923808 31233 tablet_replica.cc:333] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89: stopping tablet replica
I20251026 03:10:20.923867 31233 raft_consensus.cc:2243] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:20.923915 31233 raft_consensus.cc:2272] T c7e8905fadf4459ea1df3911271f81bf P 73c8e61facbe4889a599878198834b89 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:20.943747 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:10:20.944566 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:10:20.947332 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:20.947432 31233 tablet_replica.cc:333] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716: stopping tablet replica
I20251026 03:10:20.947490 31233 raft_consensus.cc:2243] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:20.947533 31233 raft_consensus.cc:2272] T c7e8905fadf4459ea1df3911271f81bf P 24ee1f49b0f645409c6e5d8c3c0a7716 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:20.960988 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:10:20.961759 31233 master.cc:561] Master@127.30.128.126:40143 shutting down...
I20251026 03:10:20.963770 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:20.963836 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:20.963855 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6aaf9424ecb34a39830bc01eadad044f: stopping tablet replica
I20251026 03:10:20.975649 31233 master.cc:583] Master@127.30.128.126:40143 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientBeginTxnDoesntLock (10925 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientCommitSequence
I20251026 03:10:20.994308 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:38225
I20251026 03:10:20.994467 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:20.995520 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:20.995597 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:20.995669 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:20.995587 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:20.995841 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:20.995872 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:20.995883 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448220995884 us; error 0 us; skew 500 ppm
I20251026 03:10:20.996344 31233 webserver.cc:492] Webserver started at http://127.30.128.126:43995/ using document root <none> and password file <none>
I20251026 03:10:20.996407 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:20.996465 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:20.996518 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:20.996753 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "4128e6c789f64610bcf8980f46509d64"
format_stamp: "Formatted at 2025-10-26 03:10:20 on dist-test-slave-51kf"
I20251026 03:10:20.997592 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:20.998064 4328 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:20.998214 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:20.998251 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "4128e6c789f64610bcf8980f46509d64"
format_stamp: "Formatted at 2025-10-26 03:10:20 on dist-test-slave-51kf"
I20251026 03:10:20.998291 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:21.008610 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:21.008760 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:21.010991 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:38225
I20251026 03:10:21.013247 4381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:21.013411 4380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:38225 every 8 connection(s)
I20251026 03:10:21.022027 4381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64: Bootstrap starting.
I20251026 03:10:21.022305 4381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:21.022754 4381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64: No bootstrap required, opened a new log
I20251026 03:10:21.022871 4381 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4128e6c789f64610bcf8980f46509d64" member_type: VOTER }
I20251026 03:10:21.022926 4381 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:21.022954 4381 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4128e6c789f64610bcf8980f46509d64, State: Initialized, Role: FOLLOWER
I20251026 03:10:21.023011 4381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4128e6c789f64610bcf8980f46509d64" member_type: VOTER }
I20251026 03:10:21.023044 4381 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:10:21.023068 4381 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:10:21.023094 4381 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:21.023516 4381 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4128e6c789f64610bcf8980f46509d64" member_type: VOTER }
I20251026 03:10:21.023567 4381 leader_election.cc:304] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4128e6c789f64610bcf8980f46509d64; no voters:
I20251026 03:10:21.023664 4381 leader_election.cc:290] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:10:21.023710 4384 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:21.023803 4384 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 1 LEADER]: Becoming Leader. State: Replica: 4128e6c789f64610bcf8980f46509d64, State: Running, Role: LEADER
I20251026 03:10:21.023809 4381 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:10:21.023926 4384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4128e6c789f64610bcf8980f46509d64" member_type: VOTER }
I20251026 03:10:21.024137 4385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4128e6c789f64610bcf8980f46509d64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4128e6c789f64610bcf8980f46509d64" member_type: VOTER } }
I20251026 03:10:21.024150 4386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4128e6c789f64610bcf8980f46509d64. Latest consensus state: current_term: 1 leader_uuid: "4128e6c789f64610bcf8980f46509d64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4128e6c789f64610bcf8980f46509d64" member_type: VOTER } }
I20251026 03:10:21.024258 4385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:21.024276 4386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:21.024533 4389 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:10:21.024653 4389 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:10:21.024992 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:10:21.025134 4389 catalog_manager.cc:1357] Generated new cluster ID: 708fc84aabb24c31997fa2d52df6bae6
I20251026 03:10:21.025174 4389 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:10:21.034196 4389 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:10:21.034547 4389 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:10:21.040124 4389 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64: Generated new TSK 0
I20251026 03:10:21.040182 4389 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:10:21.040652 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:21.041796 4403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:21.041811 4402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:21.041893 4405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:21.041911 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:21.042047 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:21.042075 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:21.042102 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448221042102 us; error 0 us; skew 500 ppm
I20251026 03:10:21.042613 31233 webserver.cc:492] Webserver started at http://127.30.128.65:42605/ using document root <none> and password file <none>
I20251026 03:10:21.042687 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:21.042725 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:21.042771 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:21.043015 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "a220788700174d57b59d09134a6188e4"
format_stamp: "Formatted at 2025-10-26 03:10:21 on dist-test-slave-51kf"
I20251026 03:10:21.043826 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:21.044242 4410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.044329 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.044364 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "a220788700174d57b59d09134a6188e4"
format_stamp: "Formatted at 2025-10-26 03:10:21 on dist-test-slave-51kf"
I20251026 03:10:21.044404 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:21.052471 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:21.052613 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:21.052731 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:21.052932 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:21.052964 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.052982 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:21.052995 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.055488 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:37335
I20251026 03:10:21.055541 4473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:37335 every 8 connection(s)
I20251026 03:10:21.055825 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:21.056777 4478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:21.056803 4479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:21.057045 4481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:21.057178 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:21.057304 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:21.057333 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:21.057353 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448221057352 us; error 0 us; skew 500 ppm
I20251026 03:10:21.058003 31233 webserver.cc:492] Webserver started at http://127.30.128.66:43731/ using document root <none> and password file <none>
I20251026 03:10:21.058091 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:21.058135 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:21.058192 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:21.058426 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "bb29b7b908ae44ce855ab910b2762275"
format_stamp: "Formatted at 2025-10-26 03:10:21 on dist-test-slave-51kf"
I20251026 03:10:21.059307 4474 heartbeater.cc:344] Connected to a master server at 127.30.128.126:38225
I20251026 03:10:21.059360 4474 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:21.059401 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:21.059445 4474 heartbeater.cc:507] Master 127.30.128.126:38225 requested a full tablet report, sending...
I20251026 03:10:21.059646 4345 ts_manager.cc:194] Registered new tserver with Master: a220788700174d57b59d09134a6188e4 (127.30.128.65:37335)
I20251026 03:10:21.059891 4486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.060001 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.060046 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "bb29b7b908ae44ce855ab910b2762275"
format_stamp: "Formatted at 2025-10-26 03:10:21 on dist-test-slave-51kf"
I20251026 03:10:21.060087 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:21.060340 4345 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43584
I20251026 03:10:21.067524 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:21.067669 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:21.067786 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:21.067951 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:21.067977 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.068004 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:21.068030 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.070765 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:46521
I20251026 03:10:21.070827 4549 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:46521 every 8 connection(s)
I20251026 03:10:21.071084 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:21.072010 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:21.072177 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:21.072351 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:21.073030 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:21.073168 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:21.073205 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:21.073225 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448221073224 us; error 0 us; skew 500 ppm
I20251026 03:10:21.073814 4550 heartbeater.cc:344] Connected to a master server at 127.30.128.126:38225
I20251026 03:10:21.073818 31233 webserver.cc:492] Webserver started at http://127.30.128.67:34461/ using document root <none> and password file <none>
I20251026 03:10:21.073904 4550 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:21.073931 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:21.073975 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:21.073997 4550 heartbeater.cc:507] Master 127.30.128.126:38225 requested a full tablet report, sending...
I20251026 03:10:21.074018 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:21.074213 4345 ts_manager.cc:194] Registered new tserver with Master: bb29b7b908ae44ce855ab910b2762275 (127.30.128.66:46521)
I20251026 03:10:21.074265 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "3b23a1f1d24440099a19b6e9f30298da"
format_stamp: "Formatted at 2025-10-26 03:10:21 on dist-test-slave-51kf"
I20251026 03:10:21.074522 4345 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43590
I20251026 03:10:21.075063 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:21.075419 4561 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.075511 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.075546 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "3b23a1f1d24440099a19b6e9f30298da"
format_stamp: "Formatted at 2025-10-26 03:10:21 on dist-test-slave-51kf"
I20251026 03:10:21.075587 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientCommitSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:21.087448 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:21.087634 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:21.087782 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:21.087949 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:21.087976 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.088004 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:21.088033 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.090831 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:46579
I20251026 03:10:21.090878 4624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:46579 every 8 connection(s)
I20251026 03:10:21.093273 4625 heartbeater.cc:344] Connected to a master server at 127.30.128.126:38225
I20251026 03:10:21.093322 4625 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:21.093389 4625 heartbeater.cc:507] Master 127.30.128.126:38225 requested a full tablet report, sending...
I20251026 03:10:21.093577 4345 ts_manager.cc:194] Registered new tserver with Master: 3b23a1f1d24440099a19b6e9f30298da (127.30.128.67:46579)
I20251026 03:10:21.093958 4345 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43600
I20251026 03:10:21.094295 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003277176s
I20251026 03:10:21.094589 31233 test_util.cc:276] Using random seed: 528626365
I20251026 03:10:21.098218 4345 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43604:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:10:21.098356 4345 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:10:21.102612 4438 tablet_service.cc:1505] Processing CreateTablet for tablet 63d6366d02554777bafe9c96466ddf71 (DEFAULT_TABLE table=test-workload [id=8dabd7ca9bc94db184c34128eceef953]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:21.102730 4438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d6366d02554777bafe9c96466ddf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:21.102953 4514 tablet_service.cc:1505] Processing CreateTablet for tablet 63d6366d02554777bafe9c96466ddf71 (DEFAULT_TABLE table=test-workload [id=8dabd7ca9bc94db184c34128eceef953]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:21.103058 4514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d6366d02554777bafe9c96466ddf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:21.104101 4640 tablet_bootstrap.cc:492] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: Bootstrap starting.
I20251026 03:10:21.102622 4589 tablet_service.cc:1505] Processing CreateTablet for tablet 63d6366d02554777bafe9c96466ddf71 (DEFAULT_TABLE table=test-workload [id=8dabd7ca9bc94db184c34128eceef953]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:21.104313 4641 tablet_bootstrap.cc:492] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275: Bootstrap starting.
I20251026 03:10:21.104357 4589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63d6366d02554777bafe9c96466ddf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:21.104583 4640 tablet_bootstrap.cc:654] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:21.104693 4641 tablet_bootstrap.cc:654] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:21.105187 4641 tablet_bootstrap.cc:492] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275: No bootstrap required, opened a new log
I20251026 03:10:21.105216 4641 ts_tablet_manager.cc:1403] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:21.105324 4644 tablet_bootstrap.cc:492] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: Bootstrap starting.
I20251026 03:10:21.105340 4641 raft_consensus.cc:359] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.105427 4641 raft_consensus.cc:385] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:21.105448 4641 raft_consensus.cc:740] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb29b7b908ae44ce855ab910b2762275, State: Initialized, Role: FOLLOWER
I20251026 03:10:21.105517 4641 consensus_queue.cc:260] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.105688 4641 ts_tablet_manager.cc:1434] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:10:21.105697 4640 tablet_bootstrap.cc:492] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: No bootstrap required, opened a new log
I20251026 03:10:21.105715 4550 heartbeater.cc:499] Master 127.30.128.126:38225 was elected leader, sending a full tablet report...
I20251026 03:10:21.105742 4644 tablet_bootstrap.cc:654] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:21.105816 4640 ts_tablet_manager.cc:1403] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:10:21.105923 4640 raft_consensus.cc:359] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.105952 4640 raft_consensus.cc:385] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:21.105964 4640 raft_consensus.cc:740] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a220788700174d57b59d09134a6188e4, State: Initialized, Role: FOLLOWER
I20251026 03:10:21.105999 4640 consensus_queue.cc:260] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.106128 4640 ts_tablet_manager.cc:1434] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.106199 4474 heartbeater.cc:499] Master 127.30.128.126:38225 was elected leader, sending a full tablet report...
I20251026 03:10:21.106402 4644 tablet_bootstrap.cc:492] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: No bootstrap required, opened a new log
I20251026 03:10:21.106444 4644 ts_tablet_manager.cc:1403] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:21.106560 4644 raft_consensus.cc:359] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.106598 4644 raft_consensus.cc:385] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:21.106611 4644 raft_consensus.cc:740] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b23a1f1d24440099a19b6e9f30298da, State: Initialized, Role: FOLLOWER
I20251026 03:10:21.106642 4644 consensus_queue.cc:260] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.106777 4644 ts_tablet_manager.cc:1434] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:21.106807 4625 heartbeater.cc:499] Master 127.30.128.126:38225 was elected leader, sending a full tablet report...
W20251026 03:10:21.305871 4475 tablet.cc:2378] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:10:21.309099 4648 raft_consensus.cc:493] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:10:21.309190 4648 raft_consensus.cc:515] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.309398 4648 leader_election.cc:290] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a220788700174d57b59d09134a6188e4 (127.30.128.65:37335), bb29b7b908ae44ce855ab910b2762275 (127.30.128.66:46521)
I20251026 03:10:21.311578 4448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71" candidate_uuid: "3b23a1f1d24440099a19b6e9f30298da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a220788700174d57b59d09134a6188e4" is_pre_election: true
I20251026 03:10:21.311638 4524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71" candidate_uuid: "3b23a1f1d24440099a19b6e9f30298da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb29b7b908ae44ce855ab910b2762275" is_pre_election: true
I20251026 03:10:21.311648 4448 raft_consensus.cc:2468] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b23a1f1d24440099a19b6e9f30298da in term 0.
I20251026 03:10:21.311734 4524 raft_consensus.cc:2468] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b23a1f1d24440099a19b6e9f30298da in term 0.
I20251026 03:10:21.311847 4564 leader_election.cc:304] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b23a1f1d24440099a19b6e9f30298da, a220788700174d57b59d09134a6188e4; no voters:
I20251026 03:10:21.311936 4648 raft_consensus.cc:2804] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:10:21.311976 4648 raft_consensus.cc:493] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:10:21.311997 4648 raft_consensus.cc:3060] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:21.313966 4648 raft_consensus.cc:515] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.314071 4648 leader_election.cc:290] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [CANDIDATE]: Term 1 election: Requested vote from peers a220788700174d57b59d09134a6188e4 (127.30.128.65:37335), bb29b7b908ae44ce855ab910b2762275 (127.30.128.66:46521)
I20251026 03:10:21.314250 4448 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71" candidate_uuid: "3b23a1f1d24440099a19b6e9f30298da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a220788700174d57b59d09134a6188e4"
I20251026 03:10:21.314286 4524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71" candidate_uuid: "3b23a1f1d24440099a19b6e9f30298da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb29b7b908ae44ce855ab910b2762275"
I20251026 03:10:21.314317 4448 raft_consensus.cc:3060] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:21.314321 4524 raft_consensus.cc:3060] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:21.314982 4524 raft_consensus.cc:2468] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b23a1f1d24440099a19b6e9f30298da in term 1.
I20251026 03:10:21.314982 4448 raft_consensus.cc:2468] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b23a1f1d24440099a19b6e9f30298da in term 1.
I20251026 03:10:21.315176 4564 leader_election.cc:304] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b23a1f1d24440099a19b6e9f30298da, bb29b7b908ae44ce855ab910b2762275; no voters:
I20251026 03:10:21.315256 4648 raft_consensus.cc:2804] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:21.315392 4648 raft_consensus.cc:697] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 LEADER]: Becoming Leader. State: Replica: 3b23a1f1d24440099a19b6e9f30298da, State: Running, Role: LEADER
I20251026 03:10:21.315454 4648 consensus_queue.cc:237] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:21.315965 4345 catalog_manager.cc:5649] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b23a1f1d24440099a19b6e9f30298da (127.30.128.67). New cstate: current_term: 1 leader_uuid: "3b23a1f1d24440099a19b6e9f30298da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } health_report { overall_health: HEALTHY } } }
I20251026 03:10:21.335372 4524 raft_consensus.cc:1275] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 1 FOLLOWER]: Refusing update from remote peer 3b23a1f1d24440099a19b6e9f30298da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:21.335372 4448 raft_consensus.cc:1275] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 1 FOLLOWER]: Refusing update from remote peer 3b23a1f1d24440099a19b6e9f30298da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:21.335695 4650 consensus_queue.cc:1048] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [LEADER]: Connected to new peer: Peer: permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:21.335811 4648 consensus_queue.cc:1048] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:31.736066 4670 raft_consensus.cc:993] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: : Instructing follower a220788700174d57b59d09134a6188e4 to start an election
I20251026 03:10:31.736168 4670 raft_consensus.cc:1081] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 LEADER]: Signalling peer a220788700174d57b59d09134a6188e4 to start an election
I20251026 03:10:31.736413 4447 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71"
dest_uuid: "a220788700174d57b59d09134a6188e4"
from {username='slave'} at 127.0.0.1:56392
I20251026 03:10:31.736532 4447 raft_consensus.cc:493] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:10:31.736572 4447 raft_consensus.cc:3060] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:31.737392 4447 raft_consensus.cc:515] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:31.737540 4447 leader_election.cc:290] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [CANDIDATE]: Term 2 election: Requested vote from peers bb29b7b908ae44ce855ab910b2762275 (127.30.128.66:46521), 3b23a1f1d24440099a19b6e9f30298da (127.30.128.67:46579)
I20251026 03:10:31.740103 4599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71" candidate_uuid: "a220788700174d57b59d09134a6188e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14760 } } ignore_live_leader: true dest_uuid: "3b23a1f1d24440099a19b6e9f30298da"
I20251026 03:10:31.740149 4524 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "63d6366d02554777bafe9c96466ddf71" candidate_uuid: "a220788700174d57b59d09134a6188e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 14760 } } ignore_live_leader: true dest_uuid: "bb29b7b908ae44ce855ab910b2762275"
I20251026 03:10:31.740182 4599 raft_consensus.cc:3055] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:10:31.740186 4524 raft_consensus.cc:3060] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:31.740216 4599 raft_consensus.cc:740] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3b23a1f1d24440099a19b6e9f30298da, State: Running, Role: LEADER
I20251026 03:10:31.740273 4599 consensus_queue.cc:260] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14760, Committed index: 14760, Last appended: 1.14760, Last appended by leader: 14760, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:31.740357 4599 raft_consensus.cc:3060] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:31.740876 4524 raft_consensus.cc:2468] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a220788700174d57b59d09134a6188e4 in term 2.
I20251026 03:10:31.740974 4599 raft_consensus.cc:2468] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a220788700174d57b59d09134a6188e4 in term 2.
I20251026 03:10:31.741056 4413 leader_election.cc:304] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a220788700174d57b59d09134a6188e4, bb29b7b908ae44ce855ab910b2762275; no voters:
I20251026 03:10:31.741245 4694 raft_consensus.cc:2804] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:10:31.741338 4694 raft_consensus.cc:697] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 2 LEADER]: Becoming Leader. State: Replica: a220788700174d57b59d09134a6188e4, State: Running, Role: LEADER
I20251026 03:10:31.741390 4694 consensus_queue.cc:237] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14760, Committed index: 14760, Last appended: 1.14760, Last appended by leader: 14760, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } }
I20251026 03:10:31.741803 4343 catalog_manager.cc:5649] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 reported cstate change: term changed from 1 to 2, leader changed from 3b23a1f1d24440099a19b6e9f30298da (127.30.128.67) to a220788700174d57b59d09134a6188e4 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "a220788700174d57b59d09134a6188e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220788700174d57b59d09134a6188e4" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 } health_report { overall_health: UNKNOWN } } }
I20251026 03:10:31.751411 4524 raft_consensus.cc:1275] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 2 FOLLOWER]: Refusing update from remote peer a220788700174d57b59d09134a6188e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14760. Preceding OpId from leader: term: 2 index: 14762. (index mismatch)
I20251026 03:10:31.751441 4599 raft_consensus.cc:1275] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 2 FOLLOWER]: Refusing update from remote peer a220788700174d57b59d09134a6188e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 14760. Preceding OpId from leader: term: 2 index: 14762. (index mismatch)
I20251026 03:10:31.751601 4695 consensus_queue.cc:1048] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb29b7b908ae44ce855ab910b2762275" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14761, Last known committed idx: 14760, Time since last communication: 0.000s
I20251026 03:10:31.751688 4694 consensus_queue.cc:1048] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b23a1f1d24440099a19b6e9f30298da" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14761, Last known committed idx: 14760, Time since last communication: 0.000s
W20251026 03:10:31.753803 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Cannot finalize transaction in state: OPEN
W20251026 03:10:31.754174 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction 0 already open
W20251026 03:10:31.756428 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:10:31.756748 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction 0 commit already in progress
W20251026 03:10:31.758965 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:10:31.759310 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:10:31.759634 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction 0 commit already in progress
W20251026 03:10:31.759904 4658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49882: Illegal state: Transaction metadata for transaction 0 already set
I20251026 03:10:31.760174 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:10:31.762424 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:31.762504 31233 tablet_replica.cc:333] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4: stopping tablet replica
I20251026 03:10:31.762537 31233 raft_consensus.cc:2243] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:10:31.762574 31233 raft_consensus.cc:2272] T 63d6366d02554777bafe9c96466ddf71 P a220788700174d57b59d09134a6188e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:31.775945 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:10:31.776710 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:10:31.779232 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:31.779388 31233 tablet_replica.cc:333] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275: stopping tablet replica
I20251026 03:10:31.779435 31233 raft_consensus.cc:2243] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:31.779476 31233 raft_consensus.cc:2272] T 63d6366d02554777bafe9c96466ddf71 P bb29b7b908ae44ce855ab910b2762275 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:31.799355 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:10:31.800158 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:10:31.802538 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:31.802630 31233 tablet_replica.cc:333] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da: stopping tablet replica
I20251026 03:10:31.802673 31233 raft_consensus.cc:2243] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:31.802721 31233 raft_consensus.cc:2272] T 63d6366d02554777bafe9c96466ddf71 P 3b23a1f1d24440099a19b6e9f30298da [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:31.816131 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:10:31.816816 31233 master.cc:561] Master@127.30.128.126:38225 shutting down...
I20251026 03:10:31.818898 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:31.818957 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:31.818980 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4128e6c789f64610bcf8980f46509d64: stopping tablet replica
I20251026 03:10:31.830390 31233 master.cc:583] Master@127.30.128.126:38225 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientCommitSequence (10853 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientAbortSequence
I20251026 03:10:31.848191 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:40917
I20251026 03:10:31.848342 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:31.849566 4709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.849637 4712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:31.849702 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:31.849817 4710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:31.849952 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:31.849987 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:31.849998 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448231849999 us; error 0 us; skew 500 ppm
I20251026 03:10:31.850447 31233 webserver.cc:492] Webserver started at http://127.30.128.126:39993/ using document root <none> and password file <none>
I20251026 03:10:31.850512 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:31.850556 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:31.850600 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:31.850821 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "e35b6c64845c420c9155d19bf35ed6e3"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.851593 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.851989 4717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.852125 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.852165 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "e35b6c64845c420c9155d19bf35ed6e3"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.852212 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:31.872870 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:31.873031 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:31.875434 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:40917
I20251026 03:10:31.880977 4769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:40917 every 8 connection(s)
I20251026 03:10:31.888106 4770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:31.889050 4770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3: Bootstrap starting.
I20251026 03:10:31.889302 4770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:31.889732 4770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3: No bootstrap required, opened a new log
I20251026 03:10:31.889853 4770 raft_consensus.cc:359] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35b6c64845c420c9155d19bf35ed6e3" member_type: VOTER }
I20251026 03:10:31.889921 4770 raft_consensus.cc:385] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:31.889933 4770 raft_consensus.cc:740] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35b6c64845c420c9155d19bf35ed6e3, State: Initialized, Role: FOLLOWER
I20251026 03:10:31.889966 4770 consensus_queue.cc:260] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35b6c64845c420c9155d19bf35ed6e3" member_type: VOTER }
I20251026 03:10:31.890002 4770 raft_consensus.cc:399] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:10:31.890025 4770 raft_consensus.cc:493] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:10:31.890045 4770 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:31.890429 4770 raft_consensus.cc:515] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35b6c64845c420c9155d19bf35ed6e3" member_type: VOTER }
I20251026 03:10:31.890473 4770 leader_election.cc:304] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e35b6c64845c420c9155d19bf35ed6e3; no voters:
I20251026 03:10:31.890542 4770 leader_election.cc:290] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:10:31.890604 4773 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:31.890677 4770 sys_catalog.cc:565] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:10:31.890733 4773 raft_consensus.cc:697] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 1 LEADER]: Becoming Leader. State: Replica: e35b6c64845c420c9155d19bf35ed6e3, State: Running, Role: LEADER
I20251026 03:10:31.890784 4773 consensus_queue.cc:237] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35b6c64845c420c9155d19bf35ed6e3" member_type: VOTER }
I20251026 03:10:31.890967 4774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e35b6c64845c420c9155d19bf35ed6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35b6c64845c420c9155d19bf35ed6e3" member_type: VOTER } }
I20251026 03:10:31.891026 4775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e35b6c64845c420c9155d19bf35ed6e3. Latest consensus state: current_term: 1 leader_uuid: "e35b6c64845c420c9155d19bf35ed6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e35b6c64845c420c9155d19bf35ed6e3" member_type: VOTER } }
I20251026 03:10:31.891060 4774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:31.891074 4775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:31.891319 4782 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:10:31.891433 4782 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:10:31.891911 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:10:31.892000 4782 catalog_manager.cc:1357] Generated new cluster ID: 56cc90fc6d8f4b119616ea33f8fb654e
I20251026 03:10:31.892042 4782 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:10:31.899003 4782 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:10:31.899438 4782 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:10:31.903057 4782 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3: Generated new TSK 0
I20251026 03:10:31.903136 4782 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:10:31.907436 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:31.908598 4791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.908614 4794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.908767 4792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:31.908788 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:31.908921 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:31.908954 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:31.908973 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448231908974 us; error 0 us; skew 500 ppm
I20251026 03:10:31.909507 31233 webserver.cc:492] Webserver started at http://127.30.128.65:35635/ using document root <none> and password file <none>
I20251026 03:10:31.909564 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:31.909590 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:31.909619 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:31.909823 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "358583f70cc143e6a2082615a69dcd10"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.910558 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:31.910925 4799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.911029 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.911069 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "358583f70cc143e6a2082615a69dcd10"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.911105 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:31.922780 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:31.922917 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:31.923034 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:31.923205 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:31.923230 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.923255 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:31.923275 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.925696 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:37863
I20251026 03:10:31.925724 4862 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:37863 every 8 connection(s)
I20251026 03:10:31.926054 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:31.927102 4867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.927165 4868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.927261 4870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:31.927430 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:31.927551 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:31.927583 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:31.927600 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448231927601 us; error 0 us; skew 500 ppm
I20251026 03:10:31.928272 31233 webserver.cc:492] Webserver started at http://127.30.128.66:41815/ using document root <none> and password file <none>
I20251026 03:10:31.928356 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:31.928401 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:31.928447 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:31.928720 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "4060304efe274887967f971f543dc30a"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.929615 4863 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40917
I20251026 03:10:31.929670 4863 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:31.929749 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.929759 4863 heartbeater.cc:507] Master 127.30.128.126:40917 requested a full tablet report, sending...
I20251026 03:10:31.930008 4734 ts_manager.cc:194] Registered new tserver with Master: 358583f70cc143e6a2082615a69dcd10 (127.30.128.65:37863)
I20251026 03:10:31.930287 4875 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.930409 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.930449 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "4060304efe274887967f971f543dc30a"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.930486 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:31.930671 4734 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60354
I20251026 03:10:31.951541 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:31.951689 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:31.951820 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:31.952019 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:31.952042 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.952061 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:31.952085 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.954855 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:32959
I20251026 03:10:31.955328 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:10:31.955508 4938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:32959 every 8 connection(s)
W20251026 03:10:31.956440 4941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.956446 4942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:31.956687 4945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:31.956702 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:31.956866 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:31.956894 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:31.956910 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448231956910 us; error 0 us; skew 500 ppm
I20251026 03:10:31.957471 31233 webserver.cc:492] Webserver started at http://127.30.128.67:35559/ using document root <none> and password file <none>
I20251026 03:10:31.957535 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:31.957562 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:31.957593 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:31.957808 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "065f48df49d742d1b5456b4bc406140c"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.958789 4939 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40917
I20251026 03:10:31.958858 4939 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:31.958806 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.958951 4939 heartbeater.cc:507] Master 127.30.128.126:40917 requested a full tablet report, sending...
I20251026 03:10:31.959163 4734 ts_manager.cc:194] Registered new tserver with Master: 4060304efe274887967f971f543dc30a (127.30.128.66:32959)
I20251026 03:10:31.959326 4950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.959470 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.959514 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "065f48df49d742d1b5456b4bc406140c"
format_stamp: "Formatted at 2025-10-26 03:10:31 on dist-test-slave-51kf"
I20251026 03:10:31.959558 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientAbortSequence.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:31.959615 4734 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60364
I20251026 03:10:31.962589 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:31.962759 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:31.962882 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:31.963047 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:31.963070 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.963099 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:31.963137 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:31.966081 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:45591
I20251026 03:10:31.966123 5013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:45591 every 8 connection(s)
I20251026 03:10:31.968482 5014 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40917
I20251026 03:10:31.968529 5014 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:31.968616 5014 heartbeater.cc:507] Master 127.30.128.126:40917 requested a full tablet report, sending...
I20251026 03:10:31.968823 4734 ts_manager.cc:194] Registered new tserver with Master: 065f48df49d742d1b5456b4bc406140c (127.30.128.67:45591)
I20251026 03:10:31.969123 4734 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60378
I20251026 03:10:31.969489 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003268377s
I20251026 03:10:31.969786 31233 test_util.cc:276] Using random seed: 539501561
I20251026 03:10:31.973748 4734 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60392:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:10:31.973901 4734 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:10:31.978020 4903 tablet_service.cc:1505] Processing CreateTablet for tablet 29aa61fa519f4d948ac617a7e5e244ac (DEFAULT_TABLE table=test-workload [id=58f18c20a5f841b8b226075aabe51d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:31.978147 4903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29aa61fa519f4d948ac617a7e5e244ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:31.978266 4978 tablet_service.cc:1505] Processing CreateTablet for tablet 29aa61fa519f4d948ac617a7e5e244ac (DEFAULT_TABLE table=test-workload [id=58f18c20a5f841b8b226075aabe51d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:31.978377 4978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29aa61fa519f4d948ac617a7e5e244ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:31.979415 5029 tablet_bootstrap.cc:492] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: Bootstrap starting.
I20251026 03:10:31.979503 4827 tablet_service.cc:1505] Processing CreateTablet for tablet 29aa61fa519f4d948ac617a7e5e244ac (DEFAULT_TABLE table=test-workload [id=58f18c20a5f841b8b226075aabe51d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:31.979604 4827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29aa61fa519f4d948ac617a7e5e244ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:31.979689 5030 tablet_bootstrap.cc:492] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c: Bootstrap starting.
I20251026 03:10:31.979959 5029 tablet_bootstrap.cc:654] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:31.980147 5030 tablet_bootstrap.cc:654] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:31.980545 5029 tablet_bootstrap.cc:492] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: No bootstrap required, opened a new log
I20251026 03:10:31.980583 5029 ts_tablet_manager.cc:1403] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.980757 5029 raft_consensus.cc:359] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:31.980880 5033 tablet_bootstrap.cc:492] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10: Bootstrap starting.
I20251026 03:10:31.980902 5029 raft_consensus.cc:385] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:31.980952 5029 raft_consensus.cc:740] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4060304efe274887967f971f543dc30a, State: Initialized, Role: FOLLOWER
I20251026 03:10:31.980995 5030 tablet_bootstrap.cc:492] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c: No bootstrap required, opened a new log
I20251026 03:10:31.981010 5029 consensus_queue.cc:260] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:31.981045 5030 ts_tablet_manager.cc:1403] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.981228 4939 heartbeater.cc:499] Master 127.30.128.126:40917 was elected leader, sending a full tablet report...
I20251026 03:10:31.981210 5029 ts_tablet_manager.cc:1434] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.981220 5030 raft_consensus.cc:359] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:31.981341 5030 raft_consensus.cc:385] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:31.981365 5030 raft_consensus.cc:740] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065f48df49d742d1b5456b4bc406140c, State: Initialized, Role: FOLLOWER
I20251026 03:10:31.981488 5033 tablet_bootstrap.cc:654] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:31.981519 5030 consensus_queue.cc:260] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:31.981695 5014 heartbeater.cc:499] Master 127.30.128.126:40917 was elected leader, sending a full tablet report...
I20251026 03:10:31.981686 5030 ts_tablet_manager.cc:1434] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:10:31.982133 5033 tablet_bootstrap.cc:492] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10: No bootstrap required, opened a new log
I20251026 03:10:31.982199 5033 ts_tablet_manager.cc:1403] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:31.982327 5033 raft_consensus.cc:359] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:31.982383 5033 raft_consensus.cc:385] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:31.982406 5033 raft_consensus.cc:740] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358583f70cc143e6a2082615a69dcd10, State: Initialized, Role: FOLLOWER
I20251026 03:10:31.982446 5033 consensus_queue.cc:260] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:31.982578 5033 ts_tablet_manager.cc:1434] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:10:31.982625 4863 heartbeater.cc:499] Master 127.30.128.126:40917 was elected leader, sending a full tablet report...
I20251026 03:10:32.107648 5034 raft_consensus.cc:493] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:10:32.107760 5034 raft_consensus.cc:515] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:32.107981 5034 leader_election.cc:290] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 358583f70cc143e6a2082615a69dcd10 (127.30.128.65:37863), 065f48df49d742d1b5456b4bc406140c (127.30.128.67:45591)
I20251026 03:10:32.110561 4988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac" candidate_uuid: "4060304efe274887967f971f543dc30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065f48df49d742d1b5456b4bc406140c" is_pre_election: true
I20251026 03:10:32.110564 4837 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac" candidate_uuid: "4060304efe274887967f971f543dc30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "358583f70cc143e6a2082615a69dcd10" is_pre_election: true
I20251026 03:10:32.110663 4988 raft_consensus.cc:2468] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4060304efe274887967f971f543dc30a in term 0.
I20251026 03:10:32.110687 4837 raft_consensus.cc:2468] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4060304efe274887967f971f543dc30a in term 0.
I20251026 03:10:32.110823 4879 leader_election.cc:304] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065f48df49d742d1b5456b4bc406140c, 4060304efe274887967f971f543dc30a; no voters:
I20251026 03:10:32.110910 5034 raft_consensus.cc:2804] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:10:32.110958 5034 raft_consensus.cc:493] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:10:32.110984 5034 raft_consensus.cc:3060] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:32.111655 5034 raft_consensus.cc:515] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:32.111814 5034 leader_election.cc:290] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [CANDIDATE]: Term 1 election: Requested vote from peers 358583f70cc143e6a2082615a69dcd10 (127.30.128.65:37863), 065f48df49d742d1b5456b4bc406140c (127.30.128.67:45591)
I20251026 03:10:32.111961 4837 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac" candidate_uuid: "4060304efe274887967f971f543dc30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "358583f70cc143e6a2082615a69dcd10"
I20251026 03:10:32.112007 4988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac" candidate_uuid: "4060304efe274887967f971f543dc30a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "065f48df49d742d1b5456b4bc406140c"
I20251026 03:10:32.112061 4837 raft_consensus.cc:3060] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:32.112071 4988 raft_consensus.cc:3060] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:32.112679 4837 raft_consensus.cc:2468] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4060304efe274887967f971f543dc30a in term 1.
I20251026 03:10:32.112800 4988 raft_consensus.cc:2468] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4060304efe274887967f971f543dc30a in term 1.
I20251026 03:10:32.112872 4878 leader_election.cc:304] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 358583f70cc143e6a2082615a69dcd10, 4060304efe274887967f971f543dc30a; no voters:
I20251026 03:10:32.112962 5034 raft_consensus.cc:2804] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:32.113107 5034 raft_consensus.cc:697] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 LEADER]: Becoming Leader. State: Replica: 4060304efe274887967f971f543dc30a, State: Running, Role: LEADER
I20251026 03:10:32.113155 5034 consensus_queue.cc:237] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:32.113682 4734 catalog_manager.cc:5649] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4060304efe274887967f971f543dc30a (127.30.128.66). New cstate: current_term: 1 leader_uuid: "4060304efe274887967f971f543dc30a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } health_report { overall_health: UNKNOWN } } }
I20251026 03:10:32.130882 4988 raft_consensus.cc:1275] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 1 FOLLOWER]: Refusing update from remote peer 4060304efe274887967f971f543dc30a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:32.130995 4837 raft_consensus.cc:1275] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 1 FOLLOWER]: Refusing update from remote peer 4060304efe274887967f971f543dc30a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:32.131186 5039 consensus_queue.cc:1048] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [LEADER]: Connected to new peer: Peer: permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:32.131268 5034 consensus_queue.cc:1048] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [LEADER]: Connected to new peer: Peer: permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:42.715773 5070 raft_consensus.cc:993] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: : Instructing follower 358583f70cc143e6a2082615a69dcd10 to start an election
I20251026 03:10:42.715853 5070 raft_consensus.cc:1081] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 LEADER]: Signalling peer 358583f70cc143e6a2082615a69dcd10 to start an election
I20251026 03:10:42.716005 4837 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac"
dest_uuid: "358583f70cc143e6a2082615a69dcd10"
from {username='slave'} at 127.0.0.1:60802
I20251026 03:10:42.716096 4837 raft_consensus.cc:493] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:10:42.716116 4837 raft_consensus.cc:3060] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:42.716717 4837 raft_consensus.cc:515] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:42.716868 4837 leader_election.cc:290] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [CANDIDATE]: Term 2 election: Requested vote from peers 4060304efe274887967f971f543dc30a (127.30.128.66:32959), 065f48df49d742d1b5456b4bc406140c (127.30.128.67:45591)
I20251026 03:10:42.719318 4988 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac" candidate_uuid: "358583f70cc143e6a2082615a69dcd10" candidate_term: 2 candidate_status { last_received { term: 1 index: 15194 } } ignore_live_leader: true dest_uuid: "065f48df49d742d1b5456b4bc406140c"
I20251026 03:10:42.719348 4913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "29aa61fa519f4d948ac617a7e5e244ac" candidate_uuid: "358583f70cc143e6a2082615a69dcd10" candidate_term: 2 candidate_status { last_received { term: 1 index: 15194 } } ignore_live_leader: true dest_uuid: "4060304efe274887967f971f543dc30a"
I20251026 03:10:42.719460 4913 raft_consensus.cc:3055] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:10:42.719403 4988 raft_consensus.cc:3060] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:42.719506 4913 raft_consensus.cc:740] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4060304efe274887967f971f543dc30a, State: Running, Role: LEADER
I20251026 03:10:42.719581 4913 consensus_queue.cc:260] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15194, Committed index: 15194, Last appended: 1.15194, Last appended by leader: 15194, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:42.719671 4913 raft_consensus.cc:3060] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:10:42.720115 4988 raft_consensus.cc:2468] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 358583f70cc143e6a2082615a69dcd10 in term 2.
I20251026 03:10:42.720268 4913 raft_consensus.cc:2468] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 358583f70cc143e6a2082615a69dcd10 in term 2.
I20251026 03:10:42.720276 4803 leader_election.cc:304] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 065f48df49d742d1b5456b4bc406140c, 358583f70cc143e6a2082615a69dcd10; no voters:
I20251026 03:10:42.720455 5091 raft_consensus.cc:2804] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:10:42.720644 5091 raft_consensus.cc:697] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 2 LEADER]: Becoming Leader. State: Replica: 358583f70cc143e6a2082615a69dcd10, State: Running, Role: LEADER
I20251026 03:10:42.720696 5091 consensus_queue.cc:237] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15194, Committed index: 15194, Last appended: 1.15194, Last appended by leader: 15194, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } }
I20251026 03:10:42.721117 4733 catalog_manager.cc:5649] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 reported cstate change: term changed from 1 to 2, leader changed from 4060304efe274887967f971f543dc30a (127.30.128.66) to 358583f70cc143e6a2082615a69dcd10 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "358583f70cc143e6a2082615a69dcd10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "358583f70cc143e6a2082615a69dcd10" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 37863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 } health_report { overall_health: UNKNOWN } } }
I20251026 03:10:42.733135 4913 raft_consensus.cc:1275] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 2 FOLLOWER]: Refusing update from remote peer 358583f70cc143e6a2082615a69dcd10: Log matching property violated. Preceding OpId in replica: term: 1 index: 15194. Preceding OpId from leader: term: 2 index: 15196. (index mismatch)
I20251026 03:10:42.733105 4988 raft_consensus.cc:1275] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 2 FOLLOWER]: Refusing update from remote peer 358583f70cc143e6a2082615a69dcd10: Log matching property violated. Preceding OpId in replica: term: 1 index: 15194. Preceding OpId from leader: term: 2 index: 15196. (index mismatch)
I20251026 03:10:42.733364 5091 consensus_queue.cc:1048] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4060304efe274887967f971f543dc30a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15195, Last known committed idx: 15194, Time since last communication: 0.000s
I20251026 03:10:42.733457 5092 consensus_queue.cc:1048] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "065f48df49d742d1b5456b4bc406140c" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15195, Last known committed idx: 15194, Time since last communication: 0.000s
W20251026 03:10:42.741835 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:10:42.742193 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Transaction metadata for transaction 1 already exists
W20251026 03:10:42.742542 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Cannot begin committing transaction in state: INITIALIZING
W20251026 03:10:42.742859 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Cannot begin committing transaction in state: INITIALIZING
W20251026 03:10:42.743158 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Transaction metadata for transaction 0 already exists
W20251026 03:10:42.743503 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Transaction metadata for transaction 1 already exists
W20251026 03:10:42.743831 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Transaction metadata for transaction 0 already set
W20251026 03:10:42.744151 5105 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59294: Illegal state: Transaction metadata for transaction 1 already set
I20251026 03:10:42.744467 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:10:42.746861 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:42.746968 31233 tablet_replica.cc:333] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10: stopping tablet replica
I20251026 03:10:42.747025 31233 raft_consensus.cc:2243] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:10:42.747076 31233 raft_consensus.cc:2272] T 29aa61fa519f4d948ac617a7e5e244ac P 358583f70cc143e6a2082615a69dcd10 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:42.760651 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:10:42.761413 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:10:42.764333 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:42.764461 31233 tablet_replica.cc:333] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a: stopping tablet replica
I20251026 03:10:42.764520 31233 raft_consensus.cc:2243] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:42.764585 31233 raft_consensus.cc:2272] T 29aa61fa519f4d948ac617a7e5e244ac P 4060304efe274887967f971f543dc30a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:42.783430 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:10:42.784190 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:10:42.786810 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:42.786908 31233 tablet_replica.cc:333] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c: stopping tablet replica
I20251026 03:10:42.786964 31233 raft_consensus.cc:2243] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:42.787031 31233 raft_consensus.cc:2272] T 29aa61fa519f4d948ac617a7e5e244ac P 065f48df49d742d1b5456b4bc406140c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:42.809064 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:10:42.809926 31233 master.cc:561] Master@127.30.128.126:40917 shutting down...
I20251026 03:10:42.812002 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:42.812068 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:42.812095 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P e35b6c64845c420c9155d19bf35ed6e3: stopping tablet replica
I20251026 03:10:42.823697 31233 master.cc:583] Master@127.30.128.126:40917 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientAbortSequence (10994 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun
I20251026 03:10:42.843223 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:33881
I20251026 03:10:42.843441 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:42.844563 5113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.844636 5114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.844661 5116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:42.844813 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:42.844884 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:42.844913 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:42.844925 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448242844926 us; error 0 us; skew 500 ppm
I20251026 03:10:42.845465 31233 webserver.cc:492] Webserver started at http://127.30.128.126:34287/ using document root <none> and password file <none>
I20251026 03:10:42.845556 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:42.845597 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:42.845643 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:42.845930 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "cfec0c39ac19418c93f1ec0cf944cf1e"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.846680 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:42.847038 5121 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.847136 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.847174 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "cfec0c39ac19418c93f1ec0cf944cf1e"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.847218 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:42.856999 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:42.857152 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:42.859452 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:33881
I20251026 03:10:42.861809 5173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:33881 every 8 connection(s)
I20251026 03:10:42.861925 5174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:42.862825 5174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e: Bootstrap starting.
I20251026 03:10:42.863093 5174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:42.863516 5174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e: No bootstrap required, opened a new log
I20251026 03:10:42.863641 5174 raft_consensus.cc:359] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" member_type: VOTER }
I20251026 03:10:42.863696 5174 raft_consensus.cc:385] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:42.863720 5174 raft_consensus.cc:740] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec0c39ac19418c93f1ec0cf944cf1e, State: Initialized, Role: FOLLOWER
I20251026 03:10:42.863777 5174 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" member_type: VOTER }
I20251026 03:10:42.863812 5174 raft_consensus.cc:399] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:10:42.863833 5174 raft_consensus.cc:493] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:10:42.863860 5174 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:42.864301 5174 raft_consensus.cc:515] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" member_type: VOTER }
I20251026 03:10:42.864353 5174 leader_election.cc:304] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfec0c39ac19418c93f1ec0cf944cf1e; no voters:
I20251026 03:10:42.864447 5174 leader_election.cc:290] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:10:42.864488 5177 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:42.864580 5174 sys_catalog.cc:565] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:10:42.864596 5177 raft_consensus.cc:697] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 1 LEADER]: Becoming Leader. State: Replica: cfec0c39ac19418c93f1ec0cf944cf1e, State: Running, Role: LEADER
I20251026 03:10:42.864645 5177 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" member_type: VOTER }
I20251026 03:10:42.864822 5178 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" member_type: VOTER } }
I20251026 03:10:42.864830 5179 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfec0c39ac19418c93f1ec0cf944cf1e. Latest consensus state: current_term: 1 leader_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfec0c39ac19418c93f1ec0cf944cf1e" member_type: VOTER } }
I20251026 03:10:42.864897 5178 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:42.864903 5179 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:42.865058 5181 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:10:42.865195 5181 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:10:42.865710 5181 catalog_manager.cc:1357] Generated new cluster ID: ac2661a72903495aacd6a72e9c7e73fa
I20251026 03:10:42.865749 5181 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:10:42.865898 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:10:42.879086 5181 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:10:42.879500 5181 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:10:42.890115 5181 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e: Generated new TSK 0
I20251026 03:10:42.890200 5181 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:10:42.897576 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:42.898787 5195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.898798 5198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.898838 5196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:42.898861 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:42.899137 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:42.899164 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:42.899190 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448242899190 us; error 0 us; skew 500 ppm
I20251026 03:10:42.899773 31233 webserver.cc:492] Webserver started at http://127.30.128.65:38893/ using document root <none> and password file <none>
I20251026 03:10:42.899865 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:42.899911 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:42.899962 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:42.900211 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "75dc2d334f60410d98afaaf2960374e1"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.901005 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:42.901397 5203 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.901499 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.901539 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "75dc2d334f60410d98afaaf2960374e1"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.901583 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:42.905519 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:42.905692 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:42.905840 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:42.906009 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:42.906035 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.906077 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:42.906104 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.908581 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:39205
I20251026 03:10:42.909176 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:10:42.909518 5266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:39205 every 8 connection(s)
W20251026 03:10:42.910308 5269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.910329 5272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.910524 5270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:42.910977 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:42.911072 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:42.911098 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:42.911115 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448242911115 us; error 0 us; skew 500 ppm
I20251026 03:10:42.911695 31233 webserver.cc:492] Webserver started at http://127.30.128.66:44701/ using document root <none> and password file <none>
I20251026 03:10:42.911782 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:42.911823 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:42.911883 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:42.912117 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "9afb40ae2b6b47a99493eef4f88a3780"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.912941 5267 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33881
I20251026 03:10:42.912997 5267 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:42.913051 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 03:10:42.913097 5267 heartbeater.cc:507] Master 127.30.128.126:33881 requested a full tablet report, sending...
I20251026 03:10:42.913307 5138 ts_manager.cc:194] Registered new tserver with Master: 75dc2d334f60410d98afaaf2960374e1 (127.30.128.65:39205)
I20251026 03:10:42.913558 5279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.913678 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.913724 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "9afb40ae2b6b47a99493eef4f88a3780"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.913759 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:42.913988 5138 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40498
I20251026 03:10:42.930264 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:42.930420 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:42.930533 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:42.930682 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:42.930702 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.930727 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:42.930742 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.933296 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:42539
I20251026 03:10:42.933359 5342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:42539 every 8 connection(s)
I20251026 03:10:42.933630 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:42.934803 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.934823 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:42.934855 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:42.935178 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:42.935300 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:42.935339 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:42.935369 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448242935368 us; error 0 us; skew 500 ppm
I20251026 03:10:42.935915 31233 webserver.cc:492] Webserver started at http://127.30.128.67:46547/ using document root <none> and password file <none>
I20251026 03:10:42.935992 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:42.936053 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:42.936105 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:42.936369 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "a8ac0488b37f44008099816d63711b71"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.936419 5343 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33881
I20251026 03:10:42.936471 5343 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:42.936589 5343 heartbeater.cc:507] Master 127.30.128.126:33881 requested a full tablet report, sending...
I20251026 03:10:42.936810 5138 ts_manager.cc:194] Registered new tserver with Master: 9afb40ae2b6b47a99493eef4f88a3780 (127.30.128.66:42539)
I20251026 03:10:42.937247 5138 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40508
I20251026 03:10:42.937346 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:42.937798 5354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.937919 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.937959 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "a8ac0488b37f44008099816d63711b71"
format_stamp: "Formatted at 2025-10-26 03:10:42 on dist-test-slave-51kf"
I20251026 03:10:42.938001 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:42.972692 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:42.972944 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:42.973109 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:42.973299 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:42.973322 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.973380 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:42.973407 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:42.976285 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:36251
I20251026 03:10:42.976374 5417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:36251 every 8 connection(s)
I20251026 03:10:42.979065 5418 heartbeater.cc:344] Connected to a master server at 127.30.128.126:33881
I20251026 03:10:42.979116 5418 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:42.979204 5418 heartbeater.cc:507] Master 127.30.128.126:33881 requested a full tablet report, sending...
I20251026 03:10:42.979424 5138 ts_manager.cc:194] Registered new tserver with Master: a8ac0488b37f44008099816d63711b71 (127.30.128.67:36251)
I20251026 03:10:42.979743 5138 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40512
I20251026 03:10:42.979781 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003316307s
I20251026 03:10:42.980157 31233 test_util.cc:276] Using random seed: 550511932
I20251026 03:10:42.984071 5138 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40514:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:10:42.984182 5138 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:10:42.988088 5382 tablet_service.cc:1505] Processing CreateTablet for tablet 0086ce2196f4461aa5686d7efff01cf5 (DEFAULT_TABLE table=test-workload [id=db05d4a8f00142ecae6c4c35a834355d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:42.988209 5382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0086ce2196f4461aa5686d7efff01cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:42.988585 5231 tablet_service.cc:1505] Processing CreateTablet for tablet 0086ce2196f4461aa5686d7efff01cf5 (DEFAULT_TABLE table=test-workload [id=db05d4a8f00142ecae6c4c35a834355d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:42.988683 5231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0086ce2196f4461aa5686d7efff01cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:42.989629 5433 tablet_bootstrap.cc:492] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71: Bootstrap starting.
I20251026 03:10:42.989667 5307 tablet_service.cc:1505] Processing CreateTablet for tablet 0086ce2196f4461aa5686d7efff01cf5 (DEFAULT_TABLE table=test-workload [id=db05d4a8f00142ecae6c4c35a834355d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:42.989820 5307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0086ce2196f4461aa5686d7efff01cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:42.989977 5434 tablet_bootstrap.cc:492] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1: Bootstrap starting.
I20251026 03:10:42.990175 5433 tablet_bootstrap.cc:654] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:42.990437 5434 tablet_bootstrap.cc:654] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:42.991030 5434 tablet_bootstrap.cc:492] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1: No bootstrap required, opened a new log
I20251026 03:10:42.991035 5437 tablet_bootstrap.cc:492] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780: Bootstrap starting.
I20251026 03:10:42.991077 5434 ts_tablet_manager.cc:1403] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:42.991214 5434 raft_consensus.cc:359] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:42.991281 5434 raft_consensus.cc:385] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:42.991302 5434 raft_consensus.cc:740] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75dc2d334f60410d98afaaf2960374e1, State: Initialized, Role: FOLLOWER
I20251026 03:10:42.991353 5433 tablet_bootstrap.cc:492] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71: No bootstrap required, opened a new log
I20251026 03:10:42.991369 5434 consensus_queue.cc:260] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:42.991410 5433 ts_tablet_manager.cc:1403] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:10:42.991461 5437 tablet_bootstrap.cc:654] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:42.991540 5434 ts_tablet_manager.cc:1434] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:10:42.991550 5433 raft_consensus.cc:359] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:42.991622 5267 heartbeater.cc:499] Master 127.30.128.126:33881 was elected leader, sending a full tablet report...
I20251026 03:10:42.991614 5433 raft_consensus.cc:385] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:42.991683 5433 raft_consensus.cc:740] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ac0488b37f44008099816d63711b71, State: Initialized, Role: FOLLOWER
I20251026 03:10:42.991719 5433 consensus_queue.cc:260] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:42.991868 5418 heartbeater.cc:499] Master 127.30.128.126:33881 was elected leader, sending a full tablet report...
I20251026 03:10:42.992034 5437 tablet_bootstrap.cc:492] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780: No bootstrap required, opened a new log
I20251026 03:10:42.992082 5437 ts_tablet_manager.cc:1403] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:42.992105 5433 ts_tablet_manager.cc:1434] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:10:42.992228 5437 raft_consensus.cc:359] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:42.992293 5437 raft_consensus.cc:385] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:42.992317 5437 raft_consensus.cc:740] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afb40ae2b6b47a99493eef4f88a3780, State: Initialized, Role: FOLLOWER
I20251026 03:10:42.992372 5437 consensus_queue.cc:260] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:42.992558 5437 ts_tablet_manager.cc:1434] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:10:42.992573 5343 heartbeater.cc:499] Master 127.30.128.126:33881 was elected leader, sending a full tablet report...
I20251026 03:10:43.226955 5438 raft_consensus.cc:493] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:10:43.227082 5438 raft_consensus.cc:515] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:43.227300 5438 leader_election.cc:290] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8ac0488b37f44008099816d63711b71 (127.30.128.67:36251), 9afb40ae2b6b47a99493eef4f88a3780 (127.30.128.66:42539)
I20251026 03:10:43.230103 5317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0086ce2196f4461aa5686d7efff01cf5" candidate_uuid: "75dc2d334f60410d98afaaf2960374e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afb40ae2b6b47a99493eef4f88a3780" is_pre_election: true
I20251026 03:10:43.230199 5317 raft_consensus.cc:2468] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75dc2d334f60410d98afaaf2960374e1 in term 0.
I20251026 03:10:43.230170 5392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0086ce2196f4461aa5686d7efff01cf5" candidate_uuid: "75dc2d334f60410d98afaaf2960374e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ac0488b37f44008099816d63711b71" is_pre_election: true
I20251026 03:10:43.230276 5392 raft_consensus.cc:2468] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75dc2d334f60410d98afaaf2960374e1 in term 0.
I20251026 03:10:43.230358 5206 leader_election.cc:304] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75dc2d334f60410d98afaaf2960374e1, 9afb40ae2b6b47a99493eef4f88a3780; no voters:
I20251026 03:10:43.230445 5438 raft_consensus.cc:2804] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:10:43.230472 5438 raft_consensus.cc:493] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:10:43.230501 5438 raft_consensus.cc:3060] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:43.231117 5438 raft_consensus.cc:515] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:43.231220 5438 leader_election.cc:290] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [CANDIDATE]: Term 1 election: Requested vote from peers a8ac0488b37f44008099816d63711b71 (127.30.128.67:36251), 9afb40ae2b6b47a99493eef4f88a3780 (127.30.128.66:42539)
I20251026 03:10:43.231350 5392 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0086ce2196f4461aa5686d7efff01cf5" candidate_uuid: "75dc2d334f60410d98afaaf2960374e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8ac0488b37f44008099816d63711b71"
I20251026 03:10:43.231371 5317 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0086ce2196f4461aa5686d7efff01cf5" candidate_uuid: "75dc2d334f60410d98afaaf2960374e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afb40ae2b6b47a99493eef4f88a3780"
I20251026 03:10:43.231436 5392 raft_consensus.cc:3060] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:43.231442 5317 raft_consensus.cc:3060] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:43.232151 5317 raft_consensus.cc:2468] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75dc2d334f60410d98afaaf2960374e1 in term 1.
I20251026 03:10:43.232152 5392 raft_consensus.cc:2468] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75dc2d334f60410d98afaaf2960374e1 in term 1.
I20251026 03:10:43.232340 5207 leader_election.cc:304] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75dc2d334f60410d98afaaf2960374e1, a8ac0488b37f44008099816d63711b71; no voters:
I20251026 03:10:43.232446 5438 raft_consensus.cc:2804] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:43.232571 5438 raft_consensus.cc:697] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 1 LEADER]: Becoming Leader. State: Replica: 75dc2d334f60410d98afaaf2960374e1, State: Running, Role: LEADER
I20251026 03:10:43.232632 5438 consensus_queue.cc:237] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } }
I20251026 03:10:43.233166 5137 catalog_manager.cc:5649] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75dc2d334f60410d98afaaf2960374e1 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "75dc2d334f60410d98afaaf2960374e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75dc2d334f60410d98afaaf2960374e1" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 39205 } health_report { overall_health: HEALTHY } } }
I20251026 03:10:43.277050 5317 raft_consensus.cc:1275] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 1 FOLLOWER]: Refusing update from remote peer 75dc2d334f60410d98afaaf2960374e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:43.277069 5392 raft_consensus.cc:1275] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 1 FOLLOWER]: Refusing update from remote peer 75dc2d334f60410d98afaaf2960374e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:43.277293 5443 consensus_queue.cc:1048] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ac0488b37f44008099816d63711b71" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:43.277385 5438 consensus_queue.cc:1048] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afb40ae2b6b47a99493eef4f88a3780" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:43.278532 5450 mvcc.cc:204] Tried to move back new op lower bound from 7214892004461645824 to 7214892004281389056. Current Snapshot: MvccSnapshot[applied={T|T < 7214892004461645824}]
W20251026 03:10:53.282735 5450 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42810: Illegal state: Transaction hasn't been successfully started
W20251026 03:10:53.283103 5450 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:42810: Illegal state: Transaction metadata for transaction 1 does not exist
I20251026 03:10:53.285341 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:10:53.287784 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:53.287889 31233 tablet_replica.cc:333] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1: stopping tablet replica
I20251026 03:10:53.287948 31233 raft_consensus.cc:2243] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:53.287987 31233 raft_consensus.cc:2272] T 0086ce2196f4461aa5686d7efff01cf5 P 75dc2d334f60410d98afaaf2960374e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:53.295042 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:10:53.295738 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:10:53.298209 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:53.298321 31233 tablet_replica.cc:333] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780: stopping tablet replica
I20251026 03:10:53.298378 31233 raft_consensus.cc:2243] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:53.298421 31233 raft_consensus.cc:2272] T 0086ce2196f4461aa5686d7efff01cf5 P 9afb40ae2b6b47a99493eef4f88a3780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:53.301388 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:10:53.302080 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:10:53.304425 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:10:53.304504 31233 tablet_replica.cc:333] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71: stopping tablet replica
I20251026 03:10:53.304559 31233 raft_consensus.cc:2243] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 03:10:53.304591 31233 raft_consensus.cc:2272] T 0086ce2196f4461aa5686d7efff01cf5 P a8ac0488b37f44008099816d63711b71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:53.326833 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:10:53.327551 31233 master.cc:561] Master@127.30.128.126:33881 shutting down...
I20251026 03:10:53.329411 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:10:53.329457 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:10:53.329473 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P cfec0c39ac19418c93f1ec0cf944cf1e: stopping tablet replica
I20251026 03:10:53.340955 31233 master.cc:583] Master@127.30.128.126:33881 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientErrorWhenNotBegun (10515 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientRepeatCalls
I20251026 03:10:53.358547 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:32915
I20251026 03:10:53.358659 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:10:53.360033 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:53.360072 5495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.360109 5498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.360255 5496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:53.360401 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:53.360443 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:53.360455 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448253360456 us; error 0 us; skew 500 ppm
I20251026 03:10:53.360958 31233 webserver.cc:492] Webserver started at http://127.30.128.126:37567/ using document root <none> and password file <none>
I20251026 03:10:53.361018 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:53.361050 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:53.361083 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:53.361291 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "2bef9d16cd0c4cf68e5f16c73edad646"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.362112 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:53.362462 5503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.362566 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.362607 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "2bef9d16cd0c4cf68e5f16c73edad646"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.362636 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:53.375633 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:53.375752 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:53.377832 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:32915
I20251026 03:10:53.383543 5555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:32915 every 8 connection(s)
I20251026 03:10:53.383671 5556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:53.384510 5556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646: Bootstrap starting.
I20251026 03:10:53.384805 5556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:53.385205 5556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646: No bootstrap required, opened a new log
I20251026 03:10:53.385309 5556 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" member_type: VOTER }
I20251026 03:10:53.385350 5556 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:53.385362 5556 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef9d16cd0c4cf68e5f16c73edad646, State: Initialized, Role: FOLLOWER
I20251026 03:10:53.385406 5556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" member_type: VOTER }
I20251026 03:10:53.385443 5556 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:10:53.385469 5556 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:10:53.385494 5556 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:53.385965 5556 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" member_type: VOTER }
I20251026 03:10:53.386014 5556 leader_election.cc:304] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bef9d16cd0c4cf68e5f16c73edad646; no voters:
I20251026 03:10:53.386102 5556 leader_election.cc:290] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:10:53.386161 5559 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:53.386233 5556 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:10:53.386305 5559 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 1 LEADER]: Becoming Leader. State: Replica: 2bef9d16cd0c4cf68e5f16c73edad646, State: Running, Role: LEADER
I20251026 03:10:53.386349 5559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" member_type: VOTER }
I20251026 03:10:53.386561 5560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" member_type: VOTER } }
I20251026 03:10:53.386598 5561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bef9d16cd0c4cf68e5f16c73edad646. Latest consensus state: current_term: 1 leader_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bef9d16cd0c4cf68e5f16c73edad646" member_type: VOTER } }
I20251026 03:10:53.386677 5561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:53.386831 5560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [sys.catalog]: This master's current role is: LEADER
I20251026 03:10:53.386879 5565 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:10:53.387032 5565 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:10:53.387480 5565 catalog_manager.cc:1357] Generated new cluster ID: cf5b8c8e69dc4bacafb258508cde8af9
I20251026 03:10:53.387537 5565 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:10:53.387480 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:10:53.403021 5565 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:10:53.403378 5565 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:10:53.414765 5565 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646: Generated new TSK 0
I20251026 03:10:53.414848 5565 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:10:53.419268 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:53.420332 5578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.420400 5580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:53.420426 31233 server_base.cc:1047] running on GCE node
W20251026 03:10:53.420341 5577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:53.420686 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:53.420723 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:53.420737 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448253420737 us; error 0 us; skew 500 ppm
I20251026 03:10:53.421264 31233 webserver.cc:492] Webserver started at http://127.30.128.65:43321/ using document root <none> and password file <none>
I20251026 03:10:53.421343 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:53.421388 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:53.421437 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:53.421669 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "61227403e8c64f819bf38174818c24ef"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.422468 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:53.422883 5585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.423017 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.423069 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "61227403e8c64f819bf38174818c24ef"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.423110 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:53.435478 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:53.435616 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:53.435715 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:53.435843 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:53.435865 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.435891 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:53.435909 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.438355 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:33855
I20251026 03:10:53.438390 5648 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:33855 every 8 connection(s)
I20251026 03:10:53.438657 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:53.439564 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.439610 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.439745 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:53.439800 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:53.439952 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:53.439985 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:53.439996 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448253439996 us; error 0 us; skew 500 ppm
I20251026 03:10:53.440582 31233 webserver.cc:492] Webserver started at http://127.30.128.66:39745/ using document root <none> and password file <none>
I20251026 03:10:53.440670 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:53.440713 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:53.440770 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:53.441052 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "29ae8fbcca9f4a86812e287fd144967a"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.441560 5649 heartbeater.cc:344] Connected to a master server at 127.30.128.126:32915
I20251026 03:10:53.441618 5649 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:53.441717 5649 heartbeater.cc:507] Master 127.30.128.126:32915 requested a full tablet report, sending...
I20251026 03:10:53.441931 5520 ts_manager.cc:194] Registered new tserver with Master: 61227403e8c64f819bf38174818c24ef (127.30.128.65:33855)
I20251026 03:10:53.442055 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:10:53.442508 5661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.442557 5520 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47708
I20251026 03:10:53.442641 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.442679 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "29ae8fbcca9f4a86812e287fd144967a"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.442716 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:53.455420 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:53.455569 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:53.455684 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:53.455876 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:53.455906 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.455935 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:53.455960 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.458650 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:37225
I20251026 03:10:53.458701 5724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:37225 every 8 connection(s)
I20251026 03:10:53.458946 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:10:53.459898 5729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.459950 5728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:10:53.459952 5731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:10:53.460067 31233 server_base.cc:1047] running on GCE node
I20251026 03:10:53.460227 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:10:53.460256 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:10:53.460278 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448253460277 us; error 0 us; skew 500 ppm
I20251026 03:10:53.460819 31233 webserver.cc:492] Webserver started at http://127.30.128.67:43475/ using document root <none> and password file <none>
I20251026 03:10:53.460901 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:10:53.460947 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:10:53.460989 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:10:53.461259 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "30eb353f93e84ba5ae12bd12b3f4e504"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.462083 5725 heartbeater.cc:344] Connected to a master server at 127.30.128.126:32915
I20251026 03:10:53.462137 5725 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:53.462222 5725 heartbeater.cc:507] Master 127.30.128.126:32915 requested a full tablet report, sending...
I20251026 03:10:53.462392 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:53.462414 5520 ts_manager.cc:194] Registered new tserver with Master: 29ae8fbcca9f4a86812e287fd144967a (127.30.128.66:37225)
I20251026 03:10:53.462718 5520 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47712
I20251026 03:10:53.462795 5736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.462888 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.462926 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "30eb353f93e84ba5ae12bd12b3f4e504"
format_stamp: "Formatted at 2025-10-26 03:10:53 on dist-test-slave-51kf"
I20251026 03:10:53.462965 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRepeatCalls.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:10:53.469869 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:10:53.470018 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:10:53.470139 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:10:53.470276 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:10:53.470299 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.470314 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:10:53.470337 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.473232 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:36641
I20251026 03:10:53.473285 5799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:36641 every 8 connection(s)
I20251026 03:10:53.475646 5800 heartbeater.cc:344] Connected to a master server at 127.30.128.126:32915
I20251026 03:10:53.475685 5800 heartbeater.cc:461] Registering TS with master...
I20251026 03:10:53.475759 5800 heartbeater.cc:507] Master 127.30.128.126:32915 requested a full tablet report, sending...
I20251026 03:10:53.475927 5520 ts_manager.cc:194] Registered new tserver with Master: 30eb353f93e84ba5ae12bd12b3f4e504 (127.30.128.67:36641)
I20251026 03:10:53.476224 5520 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47720
I20251026 03:10:53.476645 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003252927s
I20251026 03:10:53.476930 31233 test_util.cc:276] Using random seed: 561008706
I20251026 03:10:53.480378 5520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47734:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:10:53.480479 5520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:10:53.484525 5613 tablet_service.cc:1505] Processing CreateTablet for tablet 4cca134787e0463a8a6c40bbca863f23 (DEFAULT_TABLE table=test-workload [id=64abbc783f92404698d4b8528b6b0d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:53.484623 5613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cca134787e0463a8a6c40bbca863f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:53.485059 5689 tablet_service.cc:1505] Processing CreateTablet for tablet 4cca134787e0463a8a6c40bbca863f23 (DEFAULT_TABLE table=test-workload [id=64abbc783f92404698d4b8528b6b0d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:53.485134 5689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cca134787e0463a8a6c40bbca863f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:53.485791 5815 tablet_bootstrap.cc:492] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: Bootstrap starting.
I20251026 03:10:53.485826 5764 tablet_service.cc:1505] Processing CreateTablet for tablet 4cca134787e0463a8a6c40bbca863f23 (DEFAULT_TABLE table=test-workload [id=64abbc783f92404698d4b8528b6b0d21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:10:53.485963 5764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cca134787e0463a8a6c40bbca863f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:10:53.486325 5815 tablet_bootstrap.cc:654] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:53.486907 5815 tablet_bootstrap.cc:492] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: No bootstrap required, opened a new log
I20251026 03:10:53.486963 5815 ts_tablet_manager.cc:1403] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:53.487093 5815 raft_consensus.cc:359] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.487164 5815 raft_consensus.cc:385] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:53.487190 5815 raft_consensus.cc:740] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61227403e8c64f819bf38174818c24ef, State: Initialized, Role: FOLLOWER
I20251026 03:10:53.487246 5815 consensus_queue.cc:260] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.487293 5816 tablet_bootstrap.cc:492] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a: Bootstrap starting.
I20251026 03:10:53.487396 5815 ts_tablet_manager.cc:1434] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:10:53.487476 5649 heartbeater.cc:499] Master 127.30.128.126:32915 was elected leader, sending a full tablet report...
I20251026 03:10:53.487509 5818 tablet_bootstrap.cc:492] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: Bootstrap starting.
I20251026 03:10:53.487695 5816 tablet_bootstrap.cc:654] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:53.487890 5818 tablet_bootstrap.cc:654] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: Neither blocks nor log segments found. Creating new log.
I20251026 03:10:53.488207 5816 tablet_bootstrap.cc:492] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a: No bootstrap required, opened a new log
I20251026 03:10:53.488250 5816 ts_tablet_manager.cc:1403] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:53.488374 5816 raft_consensus.cc:359] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.488416 5818 tablet_bootstrap.cc:492] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: No bootstrap required, opened a new log
I20251026 03:10:53.488432 5816 raft_consensus.cc:385] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:53.488492 5818 ts_tablet_manager.cc:1403] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:10:53.488509 5816 raft_consensus.cc:740] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ae8fbcca9f4a86812e287fd144967a, State: Initialized, Role: FOLLOWER
I20251026 03:10:53.488559 5816 consensus_queue.cc:260] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.488613 5818 raft_consensus.cc:359] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.488662 5818 raft_consensus.cc:385] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:10:53.488687 5818 raft_consensus.cc:740] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30eb353f93e84ba5ae12bd12b3f4e504, State: Initialized, Role: FOLLOWER
I20251026 03:10:53.488705 5816 ts_tablet_manager.cc:1434] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:10:53.488734 5725 heartbeater.cc:499] Master 127.30.128.126:32915 was elected leader, sending a full tablet report...
I20251026 03:10:53.488745 5818 consensus_queue.cc:260] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.488900 5818 ts_tablet_manager.cc:1434] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:10:53.488914 5800 heartbeater.cc:499] Master 127.30.128.126:32915 was elected leader, sending a full tablet report...
W20251026 03:10:53.688691 5650 tablet.cc:2378] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:10:53.757397 5823 raft_consensus.cc:493] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:10:53.757512 5823 raft_consensus.cc:515] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.757705 5823 leader_election.cc:290] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61227403e8c64f819bf38174818c24ef (127.30.128.65:33855), 29ae8fbcca9f4a86812e287fd144967a (127.30.128.66:37225)
I20251026 03:10:53.760852 5699 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23" candidate_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ae8fbcca9f4a86812e287fd144967a" is_pre_election: true
I20251026 03:10:53.760852 5623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23" candidate_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61227403e8c64f819bf38174818c24ef" is_pre_election: true
I20251026 03:10:53.760968 5699 raft_consensus.cc:2468] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30eb353f93e84ba5ae12bd12b3f4e504 in term 0.
I20251026 03:10:53.760975 5623 raft_consensus.cc:2468] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30eb353f93e84ba5ae12bd12b3f4e504 in term 0.
I20251026 03:10:53.761145 5740 leader_election.cc:304] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30eb353f93e84ba5ae12bd12b3f4e504, 61227403e8c64f819bf38174818c24ef; no voters:
I20251026 03:10:53.761242 5823 raft_consensus.cc:2804] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:10:53.761281 5823 raft_consensus.cc:493] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:10:53.761302 5823 raft_consensus.cc:3060] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:53.761950 5823 raft_consensus.cc:515] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.762055 5823 leader_election.cc:290] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [CANDIDATE]: Term 1 election: Requested vote from peers 61227403e8c64f819bf38174818c24ef (127.30.128.65:33855), 29ae8fbcca9f4a86812e287fd144967a (127.30.128.66:37225)
I20251026 03:10:53.762181 5699 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23" candidate_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29ae8fbcca9f4a86812e287fd144967a"
I20251026 03:10:53.762231 5699 raft_consensus.cc:3060] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:53.762204 5623 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23" candidate_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61227403e8c64f819bf38174818c24ef"
I20251026 03:10:53.762272 5623 raft_consensus.cc:3060] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:10:53.762939 5699 raft_consensus.cc:2468] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30eb353f93e84ba5ae12bd12b3f4e504 in term 1.
I20251026 03:10:53.762984 5623 raft_consensus.cc:2468] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30eb353f93e84ba5ae12bd12b3f4e504 in term 1.
I20251026 03:10:53.763087 5739 leader_election.cc:304] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ae8fbcca9f4a86812e287fd144967a, 30eb353f93e84ba5ae12bd12b3f4e504; no voters:
I20251026 03:10:53.763207 5823 raft_consensus.cc:2804] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:10:53.763346 5823 raft_consensus.cc:697] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 LEADER]: Becoming Leader. State: Replica: 30eb353f93e84ba5ae12bd12b3f4e504, State: Running, Role: LEADER
I20251026 03:10:53.763393 5823 consensus_queue.cc:237] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:10:53.763954 5519 catalog_manager.cc:5649] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30eb353f93e84ba5ae12bd12b3f4e504 (127.30.128.67). New cstate: current_term: 1 leader_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } health_report { overall_health: UNKNOWN } } }
I20251026 03:10:53.838474 5699 raft_consensus.cc:1275] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 1 FOLLOWER]: Refusing update from remote peer 30eb353f93e84ba5ae12bd12b3f4e504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:53.838485 5623 raft_consensus.cc:1275] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 1 FOLLOWER]: Refusing update from remote peer 30eb353f93e84ba5ae12bd12b3f4e504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:10:53.838703 5825 consensus_queue.cc:1048] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:10:53.838781 5823 consensus_queue.cc:1048] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:04.295888 5851 raft_consensus.cc:993] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: : Instructing follower 61227403e8c64f819bf38174818c24ef to start an election
I20251026 03:11:04.295986 5851 raft_consensus.cc:1081] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 LEADER]: Signalling peer 61227403e8c64f819bf38174818c24ef to start an election
I20251026 03:11:04.296167 5622 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23"
dest_uuid: "61227403e8c64f819bf38174818c24ef"
from {username='slave'} at 127.0.0.1:39562
I20251026 03:11:04.296236 5622 raft_consensus.cc:493] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:11:04.296255 5622 raft_consensus.cc:3060] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:04.296893 5622 raft_consensus.cc:515] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:11:04.297053 5622 leader_election.cc:290] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [CANDIDATE]: Term 2 election: Requested vote from peers 30eb353f93e84ba5ae12bd12b3f4e504 (127.30.128.67:36641), 29ae8fbcca9f4a86812e287fd144967a (127.30.128.66:37225)
I20251026 03:11:04.299520 5698 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23" candidate_uuid: "61227403e8c64f819bf38174818c24ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 15332 } } ignore_live_leader: true dest_uuid: "29ae8fbcca9f4a86812e287fd144967a"
I20251026 03:11:04.299587 5698 raft_consensus.cc:3060] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:04.299561 5774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4cca134787e0463a8a6c40bbca863f23" candidate_uuid: "61227403e8c64f819bf38174818c24ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 15332 } } ignore_live_leader: true dest_uuid: "30eb353f93e84ba5ae12bd12b3f4e504"
I20251026 03:11:04.299633 5774 raft_consensus.cc:3055] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:11:04.299670 5774 raft_consensus.cc:740] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 30eb353f93e84ba5ae12bd12b3f4e504, State: Running, Role: LEADER
I20251026 03:11:04.299719 5774 consensus_queue.cc:260] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15332, Committed index: 15332, Last appended: 1.15332, Last appended by leader: 15332, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:11:04.299810 5774 raft_consensus.cc:3060] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:04.300204 5698 raft_consensus.cc:2468] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61227403e8c64f819bf38174818c24ef in term 2.
I20251026 03:11:04.300405 5774 raft_consensus.cc:2468] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61227403e8c64f819bf38174818c24ef in term 2.
I20251026 03:11:04.300392 5588 leader_election.cc:304] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ae8fbcca9f4a86812e287fd144967a, 61227403e8c64f819bf38174818c24ef; no voters:
I20251026 03:11:04.300601 5876 raft_consensus.cc:2804] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:11:04.300690 5876 raft_consensus.cc:697] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 2 LEADER]: Becoming Leader. State: Replica: 61227403e8c64f819bf38174818c24ef, State: Running, Role: LEADER
I20251026 03:11:04.300776 5876 consensus_queue.cc:237] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15332, Committed index: 15332, Last appended: 1.15332, Last appended by leader: 15332, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } }
I20251026 03:11:04.301244 5519 catalog_manager.cc:5649] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef reported cstate change: term changed from 1 to 2, leader changed from 30eb353f93e84ba5ae12bd12b3f4e504 (127.30.128.67) to 61227403e8c64f819bf38174818c24ef (127.30.128.65). New cstate: current_term: 2 leader_uuid: "61227403e8c64f819bf38174818c24ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61227403e8c64f819bf38174818c24ef" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 33855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 } health_report { overall_health: UNKNOWN } } }
I20251026 03:11:04.313876 5774 raft_consensus.cc:1275] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 2 FOLLOWER]: Refusing update from remote peer 61227403e8c64f819bf38174818c24ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 15332. Preceding OpId from leader: term: 2 index: 15334. (index mismatch)
I20251026 03:11:04.313876 5698 raft_consensus.cc:1275] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 2 FOLLOWER]: Refusing update from remote peer 61227403e8c64f819bf38174818c24ef: Log matching property violated. Preceding OpId in replica: term: 1 index: 15332. Preceding OpId from leader: term: 2 index: 15334. (index mismatch)
I20251026 03:11:04.314126 5877 consensus_queue.cc:1048] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ae8fbcca9f4a86812e287fd144967a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 37225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15333, Last known committed idx: 15332, Time since last communication: 0.000s
I20251026 03:11:04.314227 5876 consensus_queue.cc:1048] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "30eb353f93e84ba5ae12bd12b3f4e504" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15333, Last known committed idx: 15332, Time since last communication: 0.000s
W20251026 03:11:04.315476 5833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: Transaction 1 already open
W20251026 03:11:04.316743 5833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: Transaction 1 commit already in progress
W20251026 03:11:04.318001 5833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: Transaction metadata for transaction 1 already set
W20251026 03:11:04.319150 5833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: Transaction 2 already open
W20251026 03:11:04.320256 5833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: Transaction 2 commit already in progress
W20251026 03:11:04.321409 5833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: Transaction metadata for transaction 2 already set
I20251026 03:11:04.321720 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:11:04.324618 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:04.324730 31233 tablet_replica.cc:333] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef: stopping tablet replica
I20251026 03:11:04.324785 31233 raft_consensus.cc:2243] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:11:04.324833 31233 raft_consensus.cc:2272] T 4cca134787e0463a8a6c40bbca863f23 P 61227403e8c64f819bf38174818c24ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:04.340565 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:11:04.341285 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:11:04.344652 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:04.344748 31233 tablet_replica.cc:333] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a: stopping tablet replica
I20251026 03:11:04.344810 31233 raft_consensus.cc:2243] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:04.344852 31233 raft_consensus.cc:2272] T 4cca134787e0463a8a6c40bbca863f23 P 29ae8fbcca9f4a86812e287fd144967a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:04.348088 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:11:04.348807 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:11:04.352073 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:04.352162 31233 tablet_replica.cc:333] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504: stopping tablet replica
I20251026 03:11:04.352223 31233 raft_consensus.cc:2243] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:04.352265 31233 raft_consensus.cc:2272] T 4cca134787e0463a8a6c40bbca863f23 P 30eb353f93e84ba5ae12bd12b3f4e504 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:04.375304 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:11:04.376142 31233 master.cc:561] Master@127.30.128.126:32915 shutting down...
I20251026 03:11:04.378496 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:11:04.378564 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:04.378603 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2bef9d16cd0c4cf68e5f16c73edad646: stopping tablet replica
I20251026 03:11:04.390924 31233 master.cc:583] Master@127.30.128.126:32915 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientRepeatCalls (11051 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority
I20251026 03:11:04.410547 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:40697
I20251026 03:11:04.410702 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:11:04.412156 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:04.412199 5894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.412258 5891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.412290 5892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:04.412456 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:04.412495 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:04.412508 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448264412509 us; error 0 us; skew 500 ppm
I20251026 03:11:04.413020 31233 webserver.cc:492] Webserver started at http://127.30.128.126:33813/ using document root <none> and password file <none>
I20251026 03:11:04.413079 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:04.413110 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:04.413146 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:04.413378 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "510014452e7d4aa7b6bb1f4553d5d349"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.414219 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:04.414628 5899 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.414764 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.414811 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "510014452e7d4aa7b6bb1f4553d5d349"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.414845 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:04.419502 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:04.419625 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:04.421865 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:40697
I20251026 03:11:04.426949 5952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:04.430335 5951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:40697 every 8 connection(s)
I20251026 03:11:04.431002 5952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349: Bootstrap starting.
I20251026 03:11:04.431259 5952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:04.431663 5952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349: No bootstrap required, opened a new log
I20251026 03:11:04.431795 5952 raft_consensus.cc:359] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510014452e7d4aa7b6bb1f4553d5d349" member_type: VOTER }
I20251026 03:11:04.431844 5952 raft_consensus.cc:385] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:04.431869 5952 raft_consensus.cc:740] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510014452e7d4aa7b6bb1f4553d5d349, State: Initialized, Role: FOLLOWER
I20251026 03:11:04.431928 5952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510014452e7d4aa7b6bb1f4553d5d349" member_type: VOTER }
I20251026 03:11:04.431958 5952 raft_consensus.cc:399] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:11:04.431998 5952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:11:04.432027 5952 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:04.432466 5952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510014452e7d4aa7b6bb1f4553d5d349" member_type: VOTER }
I20251026 03:11:04.432519 5952 leader_election.cc:304] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510014452e7d4aa7b6bb1f4553d5d349; no voters:
I20251026 03:11:04.432614 5952 leader_election.cc:290] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:11:04.432659 5955 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:04.432754 5952 sys_catalog.cc:565] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:11:04.432834 5955 raft_consensus.cc:697] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 1 LEADER]: Becoming Leader. State: Replica: 510014452e7d4aa7b6bb1f4553d5d349, State: Running, Role: LEADER
I20251026 03:11:04.432894 5955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510014452e7d4aa7b6bb1f4553d5d349" member_type: VOTER }
I20251026 03:11:04.433135 5956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510014452e7d4aa7b6bb1f4553d5d349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510014452e7d4aa7b6bb1f4553d5d349" member_type: VOTER } }
I20251026 03:11:04.433156 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510014452e7d4aa7b6bb1f4553d5d349. Latest consensus state: current_term: 1 leader_uuid: "510014452e7d4aa7b6bb1f4553d5d349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510014452e7d4aa7b6bb1f4553d5d349" member_type: VOTER } }
I20251026 03:11:04.433199 5956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:04.433207 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:04.433472 5961 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:11:04.433619 5961 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:11:04.434118 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:11:04.434216 5961 catalog_manager.cc:1357] Generated new cluster ID: 33b9a116ac30470395b8761a423f1c41
I20251026 03:11:04.434263 5961 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:11:04.449930 5961 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:11:04.450337 5961 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:11:04.456576 5961 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349: Generated new TSK 0
I20251026 03:11:04.456658 5961 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:11:04.465986 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:04.467150 5973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.467180 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.467160 5974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:04.467341 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:04.467417 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:04.467445 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:04.467465 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448264467465 us; error 0 us; skew 500 ppm
I20251026 03:11:04.467968 31233 webserver.cc:492] Webserver started at http://127.30.128.65:32965/ using document root <none> and password file <none>
I20251026 03:11:04.468047 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:04.468096 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:04.468143 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:04.468369 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "243553ee9c0d4b7c913a7deec11d56f8"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.469098 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:04.469458 5981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.469565 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.469601 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "243553ee9c0d4b7c913a7deec11d56f8"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.469642 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:04.474812 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:04.474972 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:04.475080 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:04.475231 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:04.475255 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.475281 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:04.475306 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.477697 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:34063
I20251026 03:11:04.477758 6044 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:34063 every 8 connection(s)
I20251026 03:11:04.478070 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:04.478950 6049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.478994 6052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:04.479121 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:04.479157 6050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:04.479292 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:04.479326 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:04.479344 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448264479345 us; error 0 us; skew 500 ppm
I20251026 03:11:04.480019 31233 webserver.cc:492] Webserver started at http://127.30.128.66:35031/ using document root <none> and password file <none>
I20251026 03:11:04.480098 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:04.480141 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:04.480183 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:04.480401 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "9be1d1cffe164534a5054705af6dedca"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.481308 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:04.481508 6045 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40697
I20251026 03:11:04.481560 6045 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:04.481647 6045 heartbeater.cc:507] Master 127.30.128.126:40697 requested a full tablet report, sending...
I20251026 03:11:04.481843 6057 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.481914 5916 ts_manager.cc:194] Registered new tserver with Master: 243553ee9c0d4b7c913a7deec11d56f8 (127.30.128.65:34063)
I20251026 03:11:04.481961 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.482007 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "9be1d1cffe164534a5054705af6dedca"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.482038 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:04.482517 5916 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49094
I20251026 03:11:04.502629 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:04.502799 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:04.502938 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:04.503119 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:04.503140 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.503159 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:04.503173 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.506029 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:38719
I20251026 03:11:04.506086 6120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:38719 every 8 connection(s)
I20251026 03:11:04.506381 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:04.507405 6124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.507498 6125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:04.507436 6127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:04.507773 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:04.507881 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:04.507908 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:04.507923 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448264507924 us; error 0 us; skew 500 ppm
I20251026 03:11:04.508543 31233 webserver.cc:492] Webserver started at http://127.30.128.67:37767/ using document root <none> and password file <none>
I20251026 03:11:04.508617 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:04.508661 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:04.508697 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:04.509033 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "05c9229145d04fa29d7ee12c3f321a32"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.509318 6121 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40697
I20251026 03:11:04.509379 6121 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:04.509505 6121 heartbeater.cc:507] Master 127.30.128.126:40697 requested a full tablet report, sending...
I20251026 03:11:04.509792 5916 ts_manager.cc:194] Registered new tserver with Master: 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719)
I20251026 03:11:04.510106 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:04.510197 5916 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49106
I20251026 03:11:04.510587 6132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.510696 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.510744 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "05c9229145d04fa29d7ee12c3f321a32"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:04.510782 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:04.522821 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:04.523030 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:04.523182 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:04.523344 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:04.523370 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.523401 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:04.523424 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:04.526489 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:39151
I20251026 03:11:04.526542 6195 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:39151 every 8 connection(s)
I20251026 03:11:04.529062 6196 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40697
I20251026 03:11:04.529114 6196 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:04.529202 6196 heartbeater.cc:507] Master 127.30.128.126:40697 requested a full tablet report, sending...
I20251026 03:11:04.529402 5916 ts_manager.cc:194] Registered new tserver with Master: 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67:39151)
I20251026 03:11:04.529719 5916 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49114
I20251026 03:11:04.529922 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003274218s
I20251026 03:11:04.530256 31233 test_util.cc:276] Using random seed: 572062031
I20251026 03:11:04.534238 5916 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49116:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:11:04.534353 5916 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:11:04.538218 6085 tablet_service.cc:1505] Processing CreateTablet for tablet a95b0eddeb81461685dfddde3a16892a (DEFAULT_TABLE table=test-workload [id=288cfb3f28684bcf9cd2b3b47fcd6a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:04.538331 6085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95b0eddeb81461685dfddde3a16892a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:04.538656 6160 tablet_service.cc:1505] Processing CreateTablet for tablet a95b0eddeb81461685dfddde3a16892a (DEFAULT_TABLE table=test-workload [id=288cfb3f28684bcf9cd2b3b47fcd6a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:04.538748 6160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95b0eddeb81461685dfddde3a16892a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:04.539808 6211 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap starting.
I20251026 03:11:04.539901 6009 tablet_service.cc:1505] Processing CreateTablet for tablet a95b0eddeb81461685dfddde3a16892a (DEFAULT_TABLE table=test-workload [id=288cfb3f28684bcf9cd2b3b47fcd6a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:04.539991 6009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95b0eddeb81461685dfddde3a16892a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:04.539999 6212 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap starting.
I20251026 03:11:04.540226 6211 tablet_bootstrap.cc:654] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:04.540395 6212 tablet_bootstrap.cc:654] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:04.540802 6211 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: No bootstrap required, opened a new log
I20251026 03:11:04.540845 6211 ts_tablet_manager.cc:1403] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:04.540872 6212 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: No bootstrap required, opened a new log
I20251026 03:11:04.541014 6212 ts_tablet_manager.cc:1403] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:04.541055 6215 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8: Bootstrap starting.
I20251026 03:11:04.541159 6212 raft_consensus.cc:359] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.541188 6211 raft_consensus.cc:359] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.541235 6212 raft_consensus.cc:385] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:04.541244 6211 raft_consensus.cc:385] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:04.541311 6212 raft_consensus.cc:740] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c9229145d04fa29d7ee12c3f321a32, State: Initialized, Role: FOLLOWER
I20251026 03:11:04.541365 6211 raft_consensus.cc:740] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be1d1cffe164534a5054705af6dedca, State: Initialized, Role: FOLLOWER
I20251026 03:11:04.541430 6212 consensus_queue.cc:260] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.541450 6211 consensus_queue.cc:260] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.541508 6215 tablet_bootstrap.cc:654] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:04.541592 6211 ts_tablet_manager.cc:1434] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:04.541592 6212 ts_tablet_manager.cc:1434] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:04.541738 6121 heartbeater.cc:499] Master 127.30.128.126:40697 was elected leader, sending a full tablet report...
I20251026 03:11:04.541800 6196 heartbeater.cc:499] Master 127.30.128.126:40697 was elected leader, sending a full tablet report...
I20251026 03:11:04.542227 6215 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8: No bootstrap required, opened a new log
I20251026 03:11:04.542275 6215 ts_tablet_manager.cc:1403] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:04.542410 6215 raft_consensus.cc:359] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.542456 6215 raft_consensus.cc:385] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:04.542481 6215 raft_consensus.cc:740] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243553ee9c0d4b7c913a7deec11d56f8, State: Initialized, Role: FOLLOWER
I20251026 03:11:04.542539 6215 consensus_queue.cc:260] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.542687 6215 ts_tablet_manager.cc:1434] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:11:04.542737 6045 heartbeater.cc:499] Master 127.30.128.126:40697 was elected leader, sending a full tablet report...
I20251026 03:11:04.809947 6217 raft_consensus.cc:493] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:11:04.810074 6217 raft_consensus.cc:515] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.810256 6217 leader_election.cc:290] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719), 243553ee9c0d4b7c913a7deec11d56f8 (127.30.128.65:34063)
I20251026 03:11:04.812691 6095 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a" candidate_uuid: "05c9229145d04fa29d7ee12c3f321a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be1d1cffe164534a5054705af6dedca" is_pre_election: true
I20251026 03:11:04.812794 6095 raft_consensus.cc:2468] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05c9229145d04fa29d7ee12c3f321a32 in term 0.
I20251026 03:11:04.812863 6019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a" candidate_uuid: "05c9229145d04fa29d7ee12c3f321a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243553ee9c0d4b7c913a7deec11d56f8" is_pre_election: true
I20251026 03:11:04.812955 6019 raft_consensus.cc:2468] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05c9229145d04fa29d7ee12c3f321a32 in term 0.
I20251026 03:11:04.812963 6133 leader_election.cc:304] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c9229145d04fa29d7ee12c3f321a32, 9be1d1cffe164534a5054705af6dedca; no voters:
I20251026 03:11:04.813088 6217 raft_consensus.cc:2804] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:11:04.813125 6217 raft_consensus.cc:493] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:11:04.813146 6217 raft_consensus.cc:3060] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:04.813733 6217 raft_consensus.cc:515] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.813863 6217 leader_election.cc:290] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [CANDIDATE]: Term 1 election: Requested vote from peers 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719), 243553ee9c0d4b7c913a7deec11d56f8 (127.30.128.65:34063)
I20251026 03:11:04.813983 6095 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a" candidate_uuid: "05c9229145d04fa29d7ee12c3f321a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be1d1cffe164534a5054705af6dedca"
I20251026 03:11:04.813987 6019 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a" candidate_uuid: "05c9229145d04fa29d7ee12c3f321a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243553ee9c0d4b7c913a7deec11d56f8"
I20251026 03:11:04.814054 6095 raft_consensus.cc:3060] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:04.814059 6019 raft_consensus.cc:3060] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:04.814706 6095 raft_consensus.cc:2468] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05c9229145d04fa29d7ee12c3f321a32 in term 1.
I20251026 03:11:04.814751 6019 raft_consensus.cc:2468] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05c9229145d04fa29d7ee12c3f321a32 in term 1.
I20251026 03:11:04.814842 6133 leader_election.cc:304] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05c9229145d04fa29d7ee12c3f321a32, 9be1d1cffe164534a5054705af6dedca; no voters:
I20251026 03:11:04.814935 6217 raft_consensus.cc:2804] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:04.815073 6217 raft_consensus.cc:697] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 LEADER]: Becoming Leader. State: Replica: 05c9229145d04fa29d7ee12c3f321a32, State: Running, Role: LEADER
I20251026 03:11:04.815131 6217 consensus_queue.cc:237] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:04.815667 5914 catalog_manager.cc:5649] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67). New cstate: current_term: 1 leader_uuid: "05c9229145d04fa29d7ee12c3f321a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } health_report { overall_health: UNKNOWN } } }
I20251026 03:11:04.822861 6019 raft_consensus.cc:1275] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 1 FOLLOWER]: Refusing update from remote peer 05c9229145d04fa29d7ee12c3f321a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:04.822893 6095 raft_consensus.cc:1275] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 1 FOLLOWER]: Refusing update from remote peer 05c9229145d04fa29d7ee12c3f321a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:04.823058 6217 consensus_queue.cc:1048] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:04.823130 6221 consensus_queue.cc:1048] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:15.437326 6265 raft_consensus.cc:993] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: : Instructing follower 243553ee9c0d4b7c913a7deec11d56f8 to start an election
I20251026 03:11:15.437404 6265 raft_consensus.cc:1081] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 LEADER]: Signalling peer 243553ee9c0d4b7c913a7deec11d56f8 to start an election
I20251026 03:11:15.437587 6019 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a"
dest_uuid: "243553ee9c0d4b7c913a7deec11d56f8"
from {username='slave'} at 127.0.0.1:59442
I20251026 03:11:15.437654 6019 raft_consensus.cc:493] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:11:15.437671 6019 raft_consensus.cc:3060] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:15.438266 6019 raft_consensus.cc:515] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:15.438411 6019 leader_election.cc:290] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [CANDIDATE]: Term 2 election: Requested vote from peers 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719), 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67:39151)
I20251026 03:11:15.441612 6170 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a" candidate_uuid: "243553ee9c0d4b7c913a7deec11d56f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 15256 } } ignore_live_leader: true dest_uuid: "05c9229145d04fa29d7ee12c3f321a32"
I20251026 03:11:15.441612 6095 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a95b0eddeb81461685dfddde3a16892a" candidate_uuid: "243553ee9c0d4b7c913a7deec11d56f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 15256 } } ignore_live_leader: true dest_uuid: "9be1d1cffe164534a5054705af6dedca"
I20251026 03:11:15.441717 6170 raft_consensus.cc:3055] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:11:15.441761 6170 raft_consensus.cc:740] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05c9229145d04fa29d7ee12c3f321a32, State: Running, Role: LEADER
I20251026 03:11:15.441725 6095 raft_consensus.cc:3060] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:15.441838 6170 consensus_queue.cc:260] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15256, Committed index: 15256, Last appended: 1.15256, Last appended by leader: 15256, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:15.441931 6170 raft_consensus.cc:3060] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:15.442553 6170 raft_consensus.cc:2468] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243553ee9c0d4b7c913a7deec11d56f8 in term 2.
I20251026 03:11:15.442553 6095 raft_consensus.cc:2468] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 243553ee9c0d4b7c913a7deec11d56f8 in term 2.
I20251026 03:11:15.442742 5982 leader_election.cc:304] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243553ee9c0d4b7c913a7deec11d56f8, 9be1d1cffe164534a5054705af6dedca; no voters:
I20251026 03:11:15.442931 6271 raft_consensus.cc:2804] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:11:15.443058 6271 raft_consensus.cc:697] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 2 LEADER]: Becoming Leader. State: Replica: 243553ee9c0d4b7c913a7deec11d56f8, State: Running, Role: LEADER
I20251026 03:11:15.443123 6271 consensus_queue.cc:237] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15256, Committed index: 15256, Last appended: 1.15256, Last appended by leader: 15256, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:15.443531 5908 catalog_manager.cc:5649] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 reported cstate change: term changed from 1 to 2, leader changed from 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67) to 243553ee9c0d4b7c913a7deec11d56f8 (127.30.128.65). New cstate: current_term: 2 leader_uuid: "243553ee9c0d4b7c913a7deec11d56f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } health_report { overall_health: HEALTHY } } }
I20251026 03:11:15.450479 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:11:15.453184 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:15.453297 31233 tablet_replica.cc:333] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: stopping tablet replica
I20251026 03:11:15.453363 31233 raft_consensus.cc:2243] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:15.453428 31233 raft_consensus.cc:2272] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:15.466763 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:11:15.467561 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:11:15.470356 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:15.470443 31233 tablet_replica.cc:333] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: stopping tablet replica
I20251026 03:11:15.470490 31233 raft_consensus.cc:2243] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:15.470544 31233 raft_consensus.cc:2272] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:15.492602 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
W20251026 03:11:15.501171 5985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.128.67:39151: connect: Connection refused (error 111)
W20251026 03:11:15.501478 5982 consensus_peers.cc:597] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 -> Peer 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719): Couldn't send request to peer 9be1d1cffe164534a5054705af6dedca. Status: Network error: Client connection negotiation failed: client connection to 127.30.128.66:38719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251026 03:11:15.501513 5985 consensus_peers.cc:597] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 -> Peer 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67:39151): Couldn't send request to peer 05c9229145d04fa29d7ee12c3f321a32. Status: Network error: Client connection negotiation failed: client connection to 127.30.128.67:39151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251026 03:11:16.499656 6280 meta_cache.cc:302] tablet a95b0eddeb81461685dfddde3a16892a: replica 243553ee9c0d4b7c913a7deec11d56f8 (127.30.128.65:34063) has failed: Timed out: ParticipateInTransaction RPC to 127.30.128.65:34063 timed out after 1.000s (SENT)
I20251026 03:11:16.499965 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:16.501217 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:16.501235 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:16.501237 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:16.501379 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:16.501605 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:16.501641 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:16.501654 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448276501655 us; error 0 us; skew 500 ppm
I20251026 03:11:16.502230 31233 webserver.cc:492] Webserver started at http://127.30.128.66:35031/ using document root <none> and password file <none>
I20251026 03:11:16.502295 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:16.502319 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:11:16.502777 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:16.503296 6295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:16.503432 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:11:16.503474 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "9be1d1cffe164534a5054705af6dedca"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:16.503536 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:16.524569 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:16.524765 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:16.524922 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:16.525166 6302 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:11:16.525661 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:11:16.525694 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:16.525732 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:11:16.526093 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:11:16.526121 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:16.526137 6302 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap starting.
I20251026 03:11:16.529140 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:38719
I20251026 03:11:16.530473 6360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:38719 every 8 connection(s)
I20251026 03:11:16.533402 6361 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40697
I20251026 03:11:16.533461 6361 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:16.533560 6361 heartbeater.cc:507] Master 127.30.128.126:40697 requested a full tablet report, sending...
I20251026 03:11:16.533890 5908 ts_manager.cc:194] Re-registered known tserver with Master: 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719)
I20251026 03:11:16.534269 5908 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46796
I20251026 03:11:17.205246 6302 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap replayed 1/4 log segments. Stats: ops{read=4951 overwritten=0 applied=4947 ignored=0} inserts{seen=247300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20251026 03:11:17.534773 6361 heartbeater.cc:499] Master 127.30.128.126:40697 was elected leader, sending a full tablet report...
W20251026 03:11:17.694634 5982 consensus_peers.cc:597] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 -> Peer 9be1d1cffe164534a5054705af6dedca (127.30.128.66:38719): Couldn't send request to peer 9be1d1cffe164534a5054705af6dedca. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20251026 03:11:17.803812 5985 consensus_peers.cc:597] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 -> Peer 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67:39151): Couldn't send request to peer 05c9229145d04fa29d7ee12c3f321a32. Status: Network error: Client connection negotiation failed: client connection to 127.30.128.67:39151: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251026 03:11:17.949479 6302 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap replayed 2/4 log segments. Stats: ops{read=9896 overwritten=0 applied=9895 ignored=0} inserts{seen=494700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:11:18.710659 6302 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap replayed 3/4 log segments. Stats: ops{read=14845 overwritten=0 applied=14842 ignored=0} inserts{seen=742050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:11:18.775976 6302 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap replayed 4/4 log segments. Stats: ops{read=15256 overwritten=0 applied=15256 ignored=0} inserts{seen=762750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:11:18.776367 6302 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Bootstrap complete.
I20251026 03:11:18.781170 6302 ts_tablet_manager.cc:1403] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Time spent bootstrapping tablet: real 2.255s user 1.967s sys 0.284s
I20251026 03:11:18.781306 6302 raft_consensus.cc:359] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:18.781363 6302 raft_consensus.cc:740] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be1d1cffe164534a5054705af6dedca, State: Initialized, Role: FOLLOWER
I20251026 03:11:18.781399 6302 consensus_queue.cc:260] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15256, Last appended: 1.15256, Last appended by leader: 15256, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:18.781569 6302 ts_tablet_manager.cc:1434] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20251026 03:11:18.781849 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:18.783106 6373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:18.783243 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:18.783082 6370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:18.783053 6371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:18.783439 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:18.783483 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:18.783497 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448278783497 us; error 0 us; skew 500 ppm
I20251026 03:11:18.784143 31233 webserver.cc:492] Webserver started at http://127.30.128.67:37767/ using document root <none> and password file <none>
I20251026 03:11:18.784220 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:18.784258 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:11:18.784854 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20251026 03:11:18.785346 6378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:18.785454 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20251026 03:11:18.785497 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "05c9229145d04fa29d7ee12c3f321a32"
format_stamp: "Formatted at 2025-10-26 03:11:04 on dist-test-slave-51kf"
I20251026 03:11:18.785565 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:18.789328 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:18.789489 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:18.789613 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:18.789921 6385 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:11:18.790480 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:11:18.790515 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:18.790557 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:11:18.790923 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:11:18.790951 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:18.790961 6385 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap starting.
I20251026 03:11:18.794275 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:39151
I20251026 03:11:18.794590 6443 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:39151 every 8 connection(s)
I20251026 03:11:18.797185 6444 heartbeater.cc:344] Connected to a master server at 127.30.128.126:40697
I20251026 03:11:18.797246 6444 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:18.797377 6444 heartbeater.cc:507] Master 127.30.128.126:40697 requested a full tablet report, sending...
I20251026 03:11:18.797729 5908 ts_manager.cc:194] Re-registered known tserver with Master: 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67:39151)
I20251026 03:11:18.798095 5908 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46806
W20251026 03:11:18.824903 6454 rpcz_store.cc:267] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.0.1:53414 (ReqId={client: d32ac729a616444383100cfe92d2568e, seq_no=0, attempt_no=0}) took 3324 ms (client timeout 999 ms). Trace:
W20251026 03:11:18.825001 6454 rpcz_store.cc:269] 1026 03:11:15.500556 (+ 0us) service_pool.cc:168] Inserting onto call queue
1026 03:11:15.500584 (+ 28us) service_pool.cc:225] Handling call
1026 03:11:18.824871 (+3324287us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1026 03:11:15.500706 (+ 0us) participant_op.cc:163] PREPARE: starting
1026 03:11:15.500709 (+ 3us) participant_op.cc:199] PREPARE: finished
1026 03:11:15.500721 (+ 12us) participant_op.cc:212] START. Timestamp: 1761448275500719
1026 03:11:15.500721 (+ 0us) op_driver.cc:348] REPLICATION: starting
1026 03:11:15.500747 (+ 26us) log.cc:844] Serialized 66 byte log entry
1026 03:11:18.824502 (+3323755us) op_driver.cc:464] REPLICATION: finished
1026 03:11:18.824807 (+ 305us) participant_op.cc:265] APPLY: starting
1026 03:11:18.824855 (+ 48us) participant_op.cc:270] APPLY: finished
1026 03:11:18.824864 (+ 9us) log.cc:844] Serialized 15 byte log entry
1026 03:11:18.824866 (+ 2us) participant_op.cc:94] Released txn lock
1026 03:11:18.824867 (+ 1us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":297,"prepare.queue_time_us":78,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":84,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"replication_time_us":3323776,"thread_start_us":105,"threads_started":2}]]}
I20251026 03:11:19.443758 6385 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap replayed 1/4 log segments. Stats: ops{read=4621 overwritten=0 applied=4618 ignored=0} inserts{seen=230850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:11:19.798705 6444 heartbeater.cc:499] Master 127.30.128.126:40697 was elected leader, sending a full tablet report...
I20251026 03:11:20.132690 6385 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap replayed 2/4 log segments. Stats: ops{read=9243 overwritten=0 applied=9239 ignored=0} inserts{seen=461900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
W20251026 03:11:20.206861 5985 consensus_peers.cc:597] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 -> Peer 05c9229145d04fa29d7ee12c3f321a32 (127.30.128.67:39151): Couldn't send request to peer 05c9229145d04fa29d7ee12c3f321a32. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20251026 03:11:20.822863 6385 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap replayed 3/4 log segments. Stats: ops{read=13864 overwritten=0 applied=13861 ignored=0} inserts{seen=693000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:11:21.029116 6385 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap replayed 4/4 log segments. Stats: ops{read=15256 overwritten=0 applied=15256 ignored=0} inserts{seen=762750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:11:21.029506 6385 tablet_bootstrap.cc:492] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Bootstrap complete.
I20251026 03:11:21.034722 6385 ts_tablet_manager.cc:1403] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Time spent bootstrapping tablet: real 2.244s user 1.906s sys 0.336s
I20251026 03:11:21.034895 6385 raft_consensus.cc:359] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:21.034952 6385 raft_consensus.cc:740] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c9229145d04fa29d7ee12c3f321a32, State: Initialized, Role: FOLLOWER
I20251026 03:11:21.035009 6385 consensus_queue.cc:260] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15256, Last appended: 1.15256, Last appended by leader: 15256, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be1d1cffe164534a5054705af6dedca" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 38719 } } peers { permanent_uuid: "05c9229145d04fa29d7ee12c3f321a32" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39151 } } peers { permanent_uuid: "243553ee9c0d4b7c913a7deec11d56f8" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 34063 } }
I20251026 03:11:21.035158 6385 ts_tablet_manager.cc:1434] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:11:21.291704 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:11:21.294533 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:21.294646 31233 tablet_replica.cc:333] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8: stopping tablet replica
I20251026 03:11:21.294700 31233 raft_consensus.cc:2243] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:11:21.294778 31233 raft_consensus.cc:2272] T a95b0eddeb81461685dfddde3a16892a P 243553ee9c0d4b7c913a7deec11d56f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:21.308112 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:11:21.308904 31233 tablet_server.cc:178] TabletServer@127.30.128.66:38719 shutting down...
I20251026 03:11:21.310956 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:21.311064 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:11:21.311120 31233 raft_consensus.cc:2243] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:21.311168 31233 raft_consensus.cc:2272] T a95b0eddeb81461685dfddde3a16892a P 9be1d1cffe164534a5054705af6dedca [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:21.330080 31233 tablet_server.cc:195] TabletServer@127.30.128.66:38719 shutdown complete.
I20251026 03:11:21.330720 31233 tablet_server.cc:178] TabletServer@127.30.128.67:39151 shutting down...
I20251026 03:11:21.332578 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:21.332700 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:11:21.332765 31233 raft_consensus.cc:2243] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:21.332818 31233 raft_consensus.cc:2272] T a95b0eddeb81461685dfddde3a16892a P 05c9229145d04fa29d7ee12c3f321a32 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:21.345996 31233 tablet_server.cc:195] TabletServer@127.30.128.67:39151 shutdown complete.
I20251026 03:11:21.346617 31233 master.cc:561] Master@127.30.128.126:40697 shutting down...
I20251026 03:11:21.348788 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:11:21.348838 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:21.348855 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 510014452e7d4aa7b6bb1f4553d5d349: stopping tablet replica
I20251026 03:11:21.360637 31233 master.cc:583] Master@127.30.128.126:40697 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientTimeoutWhenNoMajority (16967 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap
I20251026 03:11:21.378367 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:37409
I20251026 03:11:21.378477 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:21.379426 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.379455 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.379472 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:21.379630 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:21.379704 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:21.379729 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:21.379746 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448281379747 us; error 0 us; skew 500 ppm
I20251026 03:11:21.380230 31233 webserver.cc:492] Webserver started at http://127.30.128.126:33599/ using document root <none> and password file <none>
I20251026 03:11:21.380297 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:21.380336 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:21.380376 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:21.380604 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "4ce72fd7240e4924b43c91f8d2ec9f77"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.381331 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:21.381670 6472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.381803 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.381842 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "4ce72fd7240e4924b43c91f8d2ec9f77"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.381880 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:21.391242 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:21.391382 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:21.393484 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:37409
I20251026 03:11:21.399238 6525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:21.399264 6524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:37409 every 8 connection(s)
I20251026 03:11:21.405727 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77: Bootstrap starting.
I20251026 03:11:21.405999 6525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:21.406419 6525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77: No bootstrap required, opened a new log
I20251026 03:11:21.406543 6525 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" member_type: VOTER }
I20251026 03:11:21.406591 6525 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:21.406605 6525 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce72fd7240e4924b43c91f8d2ec9f77, State: Initialized, Role: FOLLOWER
I20251026 03:11:21.406651 6525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" member_type: VOTER }
I20251026 03:11:21.406685 6525 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:11:21.406714 6525 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:11:21.406744 6525 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:21.407164 6525 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" member_type: VOTER }
I20251026 03:11:21.407214 6525 leader_election.cc:304] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce72fd7240e4924b43c91f8d2ec9f77; no voters:
I20251026 03:11:21.407312 6525 leader_election.cc:290] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:11:21.407351 6528 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:21.407449 6525 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:11:21.407459 6528 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce72fd7240e4924b43c91f8d2ec9f77, State: Running, Role: LEADER
I20251026 03:11:21.407578 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" member_type: VOTER }
I20251026 03:11:21.407789 6529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" member_type: VOTER } }
I20251026 03:11:21.407799 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce72fd7240e4924b43c91f8d2ec9f77. Latest consensus state: current_term: 1 leader_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce72fd7240e4924b43c91f8d2ec9f77" member_type: VOTER } }
I20251026 03:11:21.407967 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:21.408106 6536 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:11:21.408210 6529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:21.408226 6536 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:11:21.408670 6536 catalog_manager.cc:1357] Generated new cluster ID: 5bf3ab543566441f938223ac0566c212
I20251026 03:11:21.408677 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:11:21.408718 6536 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:11:21.417110 6536 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:11:21.417449 6536 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:11:21.421450 6536 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77: Generated new TSK 0
I20251026 03:11:21.421510 6536 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:11:21.424206 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:21.425261 6547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.425273 6546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:21.425379 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:21.425453 6549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:21.425546 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:21.425578 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:21.425598 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448281425598 us; error 0 us; skew 500 ppm
I20251026 03:11:21.426113 31233 webserver.cc:492] Webserver started at http://127.30.128.65:37125/ using document root <none> and password file <none>
I20251026 03:11:21.426182 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:21.426218 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:21.426249 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:21.426414 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "1314cd2aaaf04cc48783327f6a3a5394"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.427058 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:21.427381 6554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.427510 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.427553 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "1314cd2aaaf04cc48783327f6a3a5394"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.427594 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:21.445159 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:21.445288 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:21.445379 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:21.445492 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:21.445513 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.445535 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:21.445550 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.447855 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:38679
I20251026 03:11:21.447883 6617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:38679 every 8 connection(s)
I20251026 03:11:21.448123 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:21.449126 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.449154 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.449174 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:21.449306 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:21.449394 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:21.449420 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:21.449431 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448281449431 us; error 0 us; skew 500 ppm
I20251026 03:11:21.449970 31233 webserver.cc:492] Webserver started at http://127.30.128.66:34277/ using document root <none> and password file <none>
I20251026 03:11:21.450058 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:21.450098 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:21.450136 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:21.450381 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "6db2d77e455540e793d960ebafe378df"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.451123 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:21.451486 6630 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.451606 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.451643 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "6db2d77e455540e793d960ebafe378df"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.451684 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:21.454520 6618 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37409
I20251026 03:11:21.454586 6618 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:21.454674 6618 heartbeater.cc:507] Master 127.30.128.126:37409 requested a full tablet report, sending...
I20251026 03:11:21.454834 6489 ts_manager.cc:194] Registered new tserver with Master: 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:21.455276 6489 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58644
I20251026 03:11:21.467073 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:21.467204 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:21.467316 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:21.467453 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:21.467476 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.467502 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:21.467526 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.469947 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:33983
I20251026 03:11:21.469969 6693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:33983 every 8 connection(s)
I20251026 03:11:21.470227 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:21.471297 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.471318 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:21.471449 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:21.471611 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:21.471706 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:21.471743 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:21.471758 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448281471758 us; error 0 us; skew 500 ppm
I20251026 03:11:21.472199 31233 webserver.cc:492] Webserver started at http://127.30.128.67:46729/ using document root <none> and password file <none>
I20251026 03:11:21.472273 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:21.472314 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:21.472358 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:21.472571 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.472898 6694 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37409
I20251026 03:11:21.472950 6694 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:21.473030 6694 heartbeater.cc:507] Master 127.30.128.126:37409 requested a full tablet report, sending...
I20251026 03:11:21.473232 6489 ts_manager.cc:194] Registered new tserver with Master: 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983)
I20251026 03:11:21.473533 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:21.473667 6489 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58648
I20251026 03:11:21.474056 6705 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.474180 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.474224 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:21.474265 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:21.483194 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:21.483381 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:21.483529 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:21.483702 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:21.483728 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.483747 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:21.483786 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.486699 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:39809
I20251026 03:11:21.486748 6768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:39809 every 8 connection(s)
I20251026 03:11:21.488986 6769 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37409
I20251026 03:11:21.489032 6769 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:21.489120 6769 heartbeater.cc:507] Master 127.30.128.126:37409 requested a full tablet report, sending...
I20251026 03:11:21.489315 6489 ts_manager.cc:194] Registered new tserver with Master: d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809)
I20251026 03:11:21.489606 6489 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58660
I20251026 03:11:21.490104 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003255458s
I20251026 03:11:21.490374 31233 test_util.cc:276] Using random seed: 589022150
I20251026 03:11:21.493880 6489 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58672:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:11:21.493997 6489 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:11:21.498214 6733 tablet_service.cc:1505] Processing CreateTablet for tablet feca937b535d4e818a8a8ffd9bb3096a (DEFAULT_TABLE table=test-workload [id=fc7515c113864cdfbe7225b975766517]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:21.498308 6658 tablet_service.cc:1505] Processing CreateTablet for tablet feca937b535d4e818a8a8ffd9bb3096a (DEFAULT_TABLE table=test-workload [id=fc7515c113864cdfbe7225b975766517]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:21.498404 6658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feca937b535d4e818a8a8ffd9bb3096a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:21.498312 6733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feca937b535d4e818a8a8ffd9bb3096a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:21.498921 6582 tablet_service.cc:1505] Processing CreateTablet for tablet feca937b535d4e818a8a8ffd9bb3096a (DEFAULT_TABLE table=test-workload [id=fc7515c113864cdfbe7225b975766517]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:21.499004 6582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feca937b535d4e818a8a8ffd9bb3096a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:21.499552 6784 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap starting.
I20251026 03:11:21.499836 6785 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap starting.
I20251026 03:11:21.499949 6784 tablet_bootstrap.cc:654] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:21.500177 6786 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap starting.
I20251026 03:11:21.500239 6785 tablet_bootstrap.cc:654] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:21.500515 6784 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: No bootstrap required, opened a new log
I20251026 03:11:21.500557 6784 ts_tablet_manager.cc:1403] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:21.500612 6786 tablet_bootstrap.cc:654] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:21.500698 6784 raft_consensus.cc:359] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.500756 6784 raft_consensus.cc:385] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:21.500766 6785 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: No bootstrap required, opened a new log
I20251026 03:11:21.500777 6784 raft_consensus.cc:740] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db2d77e455540e793d960ebafe378df, State: Initialized, Role: FOLLOWER
I20251026 03:11:21.500797 6785 ts_tablet_manager.cc:1403] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:21.500833 6784 consensus_queue.cc:260] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.500913 6785 raft_consensus.cc:359] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.500968 6785 raft_consensus.cc:385] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:21.500972 6784 ts_tablet_manager.cc:1434] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:21.501000 6785 raft_consensus.cc:740] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25b83ae8eaf44a6b917eff8b3e2ea86, State: Initialized, Role: FOLLOWER
I20251026 03:11:21.501061 6785 consensus_queue.cc:260] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.501082 6786 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: No bootstrap required, opened a new log
I20251026 03:11:21.501065 6694 heartbeater.cc:499] Master 127.30.128.126:37409 was elected leader, sending a full tablet report...
I20251026 03:11:21.501127 6786 ts_tablet_manager.cc:1403] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:21.501227 6785 ts_tablet_manager.cc:1434] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:11:21.501299 6769 heartbeater.cc:499] Master 127.30.128.126:37409 was elected leader, sending a full tablet report...
I20251026 03:11:21.501462 6786 raft_consensus.cc:359] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.501574 6786 raft_consensus.cc:385] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:21.501636 6786 raft_consensus.cc:740] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1314cd2aaaf04cc48783327f6a3a5394, State: Initialized, Role: FOLLOWER
I20251026 03:11:21.501677 6786 consensus_queue.cc:260] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.501848 6786 ts_tablet_manager.cc:1434] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:21.501865 6618 heartbeater.cc:499] Master 127.30.128.126:37409 was elected leader, sending a full tablet report...
I20251026 03:11:21.856782 6792 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:11:21.856884 6792 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.857080 6792 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983), d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809)
I20251026 03:11:21.859340 6743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "1314cd2aaaf04cc48783327f6a3a5394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" is_pre_election: true
I20251026 03:11:21.859431 6743 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1314cd2aaaf04cc48783327f6a3a5394 in term 0.
I20251026 03:11:21.859450 6668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "1314cd2aaaf04cc48783327f6a3a5394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db2d77e455540e793d960ebafe378df" is_pre_election: true
I20251026 03:11:21.859560 6668 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1314cd2aaaf04cc48783327f6a3a5394 in term 0.
I20251026 03:11:21.859580 6557 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1314cd2aaaf04cc48783327f6a3a5394, d25b83ae8eaf44a6b917eff8b3e2ea86; no voters:
I20251026 03:11:21.859687 6792 raft_consensus.cc:2804] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:11:21.859721 6792 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:11:21.859742 6792 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:21.860329 6792 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.860435 6792 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 1 election: Requested vote from peers 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983), d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809)
I20251026 03:11:21.860581 6743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "1314cd2aaaf04cc48783327f6a3a5394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86"
I20251026 03:11:21.860639 6743 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:21.860579 6668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "1314cd2aaaf04cc48783327f6a3a5394" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db2d77e455540e793d960ebafe378df"
I20251026 03:11:21.860706 6668 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:21.861287 6743 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1314cd2aaaf04cc48783327f6a3a5394 in term 1.
I20251026 03:11:21.861408 6668 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1314cd2aaaf04cc48783327f6a3a5394 in term 1.
I20251026 03:11:21.861449 6557 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1314cd2aaaf04cc48783327f6a3a5394, d25b83ae8eaf44a6b917eff8b3e2ea86; no voters:
I20251026 03:11:21.861516 6792 raft_consensus.cc:2804] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:21.861640 6792 raft_consensus.cc:697] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 LEADER]: Becoming Leader. State: Replica: 1314cd2aaaf04cc48783327f6a3a5394, State: Running, Role: LEADER
I20251026 03:11:21.861689 6792 consensus_queue.cc:237] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:21.862237 6488 catalog_manager.cc:5649] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "1314cd2aaaf04cc48783327f6a3a5394" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } health_report { overall_health: HEALTHY } } }
I20251026 03:11:21.943089 6668 raft_consensus.cc:1275] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 1 FOLLOWER]: Refusing update from remote peer 1314cd2aaaf04cc48783327f6a3a5394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:21.943055 6743 raft_consensus.cc:1275] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Refusing update from remote peer 1314cd2aaaf04cc48783327f6a3a5394: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:21.943331 6792 consensus_queue.cc:1048] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:21.943413 6794 consensus_queue.cc:1048] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:21.945879 6802 mvcc.cc:204] Tried to move back new op lower bound from 7214892162838618112 to 7214892162506117120. Current Snapshot: MvccSnapshot[applied={T|T < 7214892162837524480}]
I20251026 03:11:31.944686 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:11:31.948128 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:31.948246 31233 tablet_replica.cc:333] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: stopping tablet replica
I20251026 03:11:31.948303 31233 raft_consensus.cc:2243] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:11:31.948357 31233 raft_consensus.cc:2272] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 03:11:31.953876 6841 meta_cache.cc:302] tablet feca937b535d4e818a8a8ffd9bb3096a: replica 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679) has failed: Network error: Client connection negotiation failed: client connection to 127.30.128.65:38679: connect: Connection refused (error 111)
W20251026 03:11:31.958977 6658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35198: Illegal state: replica 6db2d77e455540e793d960ebafe378df is not leader of this config: current role FOLLOWER
I20251026 03:11:31.961653 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:11:31.962455 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:11:31.963653 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:31.963733 6852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:31.963783 6850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:31.963793 6849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:31.964010 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:31.964046 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:31.964058 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448291964059 us; error 0 us; skew 500 ppm
I20251026 03:11:31.964630 31233 webserver.cc:492] Webserver started at http://127.30.128.65:37125/ using document root <none> and password file <none>
I20251026 03:11:31.964716 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:31.964762 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:11:31.965278 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20251026 03:11:31.965823 6858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:31.965970 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:31.966012 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "1314cd2aaaf04cc48783327f6a3a5394"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:31.966060 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251026 03:11:31.967217 6733 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59288: Illegal state: replica d25b83ae8eaf44a6b917eff8b3e2ea86 is not leader of this config: current role FOLLOWER
I20251026 03:11:31.972963 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:31.973100 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:31.973210 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:31.973417 6865 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:11:31.973877 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:11:31.973912 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:31.973932 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:11:31.974241 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:11:31.974268 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:31.974275 6865 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap starting.
I20251026 03:11:31.976985 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:38679
W20251026 03:11:31.981210 6658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35198: Illegal state: replica 6db2d77e455540e793d960ebafe378df is not leader of this config: current role FOLLOWER
I20251026 03:11:31.984270 6923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:38679 every 8 connection(s)
I20251026 03:11:31.986943 6924 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37409
I20251026 03:11:31.987037 6924 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:31.987135 6924 heartbeater.cc:507] Master 127.30.128.126:37409 requested a full tablet report, sending...
I20251026 03:11:31.987416 6488 ts_manager.cc:194] Re-registered known tserver with Master: 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:31.987754 6488 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34874
W20251026 03:11:31.989650 6733 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:59288: Illegal state: replica d25b83ae8eaf44a6b917eff8b3e2ea86 is not leader of this config: current role FOLLOWER
I20251026 03:11:32.591197 6865 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap replayed 1/4 log segments. Stats: ops{read=4621 overwritten=0 applied=4619 ignored=0} inserts{seen=230900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:11:32.988269 6924 heartbeater.cc:499] Master 127.30.128.126:37409 was elected leader, sending a full tablet report...
I20251026 03:11:33.261459 6865 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap replayed 2/4 log segments. Stats: ops{read=9241 overwritten=0 applied=9237 ignored=0} inserts{seen=461800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20251026 03:11:33.434682 6930 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1314cd2aaaf04cc48783327f6a3a5394)
I20251026 03:11:33.434791 6930 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:33.434990 6930 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983), 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:33.435369 6933 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1314cd2aaaf04cc48783327f6a3a5394)
I20251026 03:11:33.435432 6933 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:33.435827 6933 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809), 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:33.438401 6668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" candidate_term: 2 candidate_status { last_received { term: 1 index: 15312 } } ignore_live_leader: false dest_uuid: "6db2d77e455540e793d960ebafe378df" is_pre_election: true
I20251026 03:11:33.438472 6668 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d25b83ae8eaf44a6b917eff8b3e2ea86 in term 1.
I20251026 03:11:33.438661 6707 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db2d77e455540e793d960ebafe378df, d25b83ae8eaf44a6b917eff8b3e2ea86; no voters:
I20251026 03:11:33.438761 6930 raft_consensus.cc:2804] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 03:11:33.438814 6930 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1314cd2aaaf04cc48783327f6a3a5394)
I20251026 03:11:33.438836 6930 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:33.439436 6898 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" candidate_term: 2 candidate_status { last_received { term: 1 index: 15312 } } ignore_live_leader: false dest_uuid: "1314cd2aaaf04cc48783327f6a3a5394" is_pre_election: true
I20251026 03:11:33.439620 6930 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
W20251026 03:11:33.439644 6707 leader_election.cc:343] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679): Illegal state: must be running to vote when last-logged opid is not known
I20251026 03:11:33.439724 6930 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [CANDIDATE]: Term 2 election: Requested vote from peers 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983), 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:33.439844 6898 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" candidate_term: 2 candidate_status { last_received { term: 1 index: 15312 } } ignore_live_leader: false dest_uuid: "1314cd2aaaf04cc48783327f6a3a5394"
I20251026 03:11:33.439934 6668 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" candidate_term: 2 candidate_status { last_received { term: 1 index: 15312 } } ignore_live_leader: false dest_uuid: "6db2d77e455540e793d960ebafe378df"
I20251026 03:11:33.439982 6668 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 1 FOLLOWER]: Advancing to term 2
W20251026 03:11:33.440065 6707 leader_election.cc:343] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679): Illegal state: must be running to vote when last-logged opid is not known
I20251026 03:11:33.440554 6743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "6db2d77e455540e793d960ebafe378df" candidate_term: 2 candidate_status { last_received { term: 1 index: 15312 } } ignore_live_leader: false dest_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" is_pre_election: true
I20251026 03:11:33.440635 6743 raft_consensus.cc:2393] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6db2d77e455540e793d960ebafe378df in current term 2: Already voted for candidate d25b83ae8eaf44a6b917eff8b3e2ea86 in this term.
I20251026 03:11:33.440670 6668 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d25b83ae8eaf44a6b917eff8b3e2ea86 in term 2.
I20251026 03:11:33.440712 6898 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "6db2d77e455540e793d960ebafe378df" candidate_term: 2 candidate_status { last_received { term: 1 index: 15312 } } ignore_live_leader: false dest_uuid: "1314cd2aaaf04cc48783327f6a3a5394" is_pre_election: true
I20251026 03:11:33.440802 6707 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6db2d77e455540e793d960ebafe378df, d25b83ae8eaf44a6b917eff8b3e2ea86; no voters: 1314cd2aaaf04cc48783327f6a3a5394
W20251026 03:11:33.440877 6632 leader_election.cc:343] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679): Illegal state: must be running to vote when last-logged opid is not known
I20251026 03:11:33.440897 6930 raft_consensus.cc:2804] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:11:33.440913 6632 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6db2d77e455540e793d960ebafe378df; no voters: 1314cd2aaaf04cc48783327f6a3a5394, d25b83ae8eaf44a6b917eff8b3e2ea86
I20251026 03:11:33.440972 6933 raft_consensus.cc:2749] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251026 03:11:33.440997 6930 raft_consensus.cc:697] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 LEADER]: Becoming Leader. State: Replica: d25b83ae8eaf44a6b917eff8b3e2ea86, State: Running, Role: LEADER
I20251026 03:11:33.441041 6930 consensus_queue.cc:237] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15312, Committed index: 15312, Last appended: 1.15312, Last appended by leader: 15312, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:33.441485 6488 catalog_manager.cc:5649] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 reported cstate change: term changed from 1 to 2, leader changed from 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65) to d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67). New cstate: current_term: 2 leader_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } health_report { overall_health: UNKNOWN } } }
I20251026 03:11:33.899430 6668 raft_consensus.cc:1275] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Refusing update from remote peer d25b83ae8eaf44a6b917eff8b3e2ea86: Log matching property violated. Preceding OpId in replica: term: 1 index: 15312. Preceding OpId from leader: term: 2 index: 15313. (index mismatch)
I20251026 03:11:33.899683 6930 consensus_queue.cc:1048] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15313, Last known committed idx: 15312, Time since last communication: 0.000s
W20251026 03:11:33.901005 6707 consensus_peers.cc:597] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 -> Peer 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679): Couldn't send request to peer 1314cd2aaaf04cc48783327f6a3a5394. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 03:11:33.991660 6865 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap replayed 3/4 log segments. Stats: ops{read=13861 overwritten=0 applied=13860 ignored=0} inserts{seen=692950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:11:34.217394 6865 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap replayed 4/4 log segments. Stats: ops{read=15312 overwritten=0 applied=15312 ignored=0} inserts{seen=765550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:11:34.217823 6865 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Bootstrap complete.
I20251026 03:11:34.222950 6865 ts_tablet_manager.cc:1403] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Time spent bootstrapping tablet: real 2.249s user 1.936s sys 0.307s
I20251026 03:11:34.223091 6865 raft_consensus.cc:359] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:34.223165 6865 raft_consensus.cc:740] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1314cd2aaaf04cc48783327f6a3a5394, State: Initialized, Role: FOLLOWER
I20251026 03:11:34.223214 6865 consensus_queue.cc:260] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15312, Last appended: 1.15312, Last appended by leader: 15312, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:34.223364 6865 ts_tablet_manager.cc:1434] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20251026 03:11:34.254050 6888 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58542: Illegal state: replica 1314cd2aaaf04cc48783327f6a3a5394 is not leader of this config: current role FOLLOWER
W20251026 03:11:34.320847 6658 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35198: Illegal state: replica 6db2d77e455540e793d960ebafe378df is not leader of this config: current role FOLLOWER
I20251026 03:11:34.451172 6898 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:34.723567 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
W20251026 03:11:34.723992 6707 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 10 similar messages]
W20251026 03:11:34.725255 6707 consensus_peers.cc:597] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 -> Peer 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983): Couldn't send request to peer 6db2d77e455540e793d960ebafe378df. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251026 03:11:34.727563 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:34.727648 31233 tablet_replica.cc:333] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: stopping tablet replica
I20251026 03:11:34.727699 31233 raft_consensus.cc:2243] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:34.727756 31233 pending_rounds.cc:70] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Trying to abort 1 pending ops.
I20251026 03:11:34.727771 31233 pending_rounds.cc:77] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Aborting op as it isn't in flight: id { term: 2 index: 15759 } timestamp: 7214892215186489344 op_type: PARTICIPANT_OP participant_request { tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" op { type: BEGIN_COMMIT txn_id: 148 } }
I20251026 03:11:34.727833 31233 raft_consensus.cc:2272] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:34.732095 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:11:34.733309 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:34.734483 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:34.734485 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:34.734485 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:34.734999 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:34.735105 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:34.735138 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:34.735164 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448294735164 us; error 0 us; skew 500 ppm
I20251026 03:11:34.735795 31233 webserver.cc:492] Webserver started at http://127.30.128.66:34277/ using document root <none> and password file <none>
I20251026 03:11:34.735888 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:34.735916 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:11:34.736493 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:34.737083 6970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:34.737212 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:34.737257 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "6db2d77e455540e793d960ebafe378df"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:34.737305 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:34.748806 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:34.748975 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:34.749104 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:34.749398 6977 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:11:34.750219 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:11:34.750260 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:34.750298 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:11:34.750741 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:11:34.750770 6977 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap starting.
I20251026 03:11:34.750782 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:34.754035 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:33983
I20251026 03:11:34.755072 7035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:33983 every 8 connection(s)
I20251026 03:11:34.774178 7036 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37409
I20251026 03:11:34.774257 7036 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:34.774371 7036 heartbeater.cc:507] Master 127.30.128.126:37409 requested a full tablet report, sending...
I20251026 03:11:34.774749 6488 ts_manager.cc:194] Re-registered known tserver with Master: 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983)
I20251026 03:11:34.775288 6488 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34880
I20251026 03:11:35.477475 6977 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap replayed 1/4 log segments. Stats: ops{read=4939 overwritten=0 applied=4937 ignored=0} inserts{seen=246800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:11:35.775877 7036 heartbeater.cc:499] Master 127.30.128.126:37409 was elected leader, sending a full tablet report...
I20251026 03:11:36.237082 6977 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap replayed 2/4 log segments. Stats: ops{read=9869 overwritten=0 applied=9867 ignored=0} inserts{seen=493300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:11:37.039795 6977 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap replayed 3/4 log segments. Stats: ops{read=14810 overwritten=0 applied=14809 ignored=0} inserts{seen=740400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:11:37.134272 6977 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap replayed 4/4 log segments. Stats: ops{read=15759 overwritten=0 applied=15758 ignored=0} inserts{seen=765550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:11:37.134740 6977 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Bootstrap complete.
I20251026 03:11:37.139715 6977 ts_tablet_manager.cc:1403] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Time spent bootstrapping tablet: real 2.389s user 2.073s sys 0.308s
I20251026 03:11:37.139900 6977 raft_consensus.cc:359] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:37.140085 6977 raft_consensus.cc:740] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db2d77e455540e793d960ebafe378df, State: Initialized, Role: FOLLOWER
I20251026 03:11:37.140168 6977 consensus_queue.cc:260] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15758, Last appended: 2.15759, Last appended by leader: 15759, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:37.140304 6977 ts_tablet_manager.cc:1434] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 03:11:37.640515 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:11:37.643392 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:37.643524 31233 tablet_replica.cc:333] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: stopping tablet replica
I20251026 03:11:37.643589 31233 raft_consensus.cc:2243] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:11:37.643664 31233 raft_consensus.cc:2272] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251026 03:11:37.656322 6842 meta_cache.cc:302] tablet feca937b535d4e818a8a8ffd9bb3096a: replica d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809) has failed: Network error: Client connection negotiation failed: client connection to 127.30.128.67:39809: connect: Connection refused (error 111)
I20251026 03:11:37.661689 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:11:37.663192 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:37.664243 7070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:37.664281 7069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:37.664393 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:37.664347 7072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:37.664549 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:37.664592 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:37.664614 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448297664614 us; error 0 us; skew 500 ppm
I20251026 03:11:37.665334 31233 webserver.cc:492] Webserver started at http://127.30.128.67:46729/ using document root <none> and password file <none>
I20251026 03:11:37.665421 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:37.665458 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:11:37.666055 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:11:37.666587 7078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:37.666708 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:11:37.666790 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86"
format_stamp: "Formatted at 2025-10-26 03:11:21 on dist-test-slave-51kf"
I20251026 03:11:37.666849 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251026 03:11:37.667104 6992 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34586: Illegal state: replica 6db2d77e455540e793d960ebafe378df is not leader of this config: current role FOLLOWER
W20251026 03:11:37.676614 6888 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58542: Illegal state: replica 1314cd2aaaf04cc48783327f6a3a5394 is not leader of this config: current role FOLLOWER
I20251026 03:11:37.679688 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:37.679868 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:37.680015 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:37.680284 7085 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:11:37.680823 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:11:37.680853 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:37.680876 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:11:37.681254 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:11:37.681281 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:37.681291 7085 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap starting.
I20251026 03:11:37.684013 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:39809
I20251026 03:11:37.687491 7143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:39809 every 8 connection(s)
I20251026 03:11:37.694798 7144 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37409
I20251026 03:11:37.694865 7144 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:37.694976 7144 heartbeater.cc:507] Master 127.30.128.126:37409 requested a full tablet report, sending...
I20251026 03:11:37.695257 6488 ts_manager.cc:194] Re-registered known tserver with Master: d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809)
I20251026 03:11:37.695755 6488 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58312
I20251026 03:11:38.382128 7085 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap replayed 1/4 log segments. Stats: ops{read=4947 overwritten=0 applied=4944 ignored=0} inserts{seen=247150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:11:38.696257 7144 heartbeater.cc:499] Master 127.30.128.126:37409 was elected leader, sending a full tablet report...
I20251026 03:11:39.141367 7151 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d25b83ae8eaf44a6b917eff8b3e2ea86)
I20251026 03:11:39.141477 7151 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:39.141530 7152 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d25b83ae8eaf44a6b917eff8b3e2ea86)
I20251026 03:11:39.141637 7152 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:39.142192 7152 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809), 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:39.142220 7151 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6db2d77e455540e793d960ebafe378df (127.30.128.66:33983), d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809)
I20251026 03:11:39.145666 7118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "1314cd2aaaf04cc48783327f6a3a5394" candidate_term: 3 candidate_status { last_received { term: 2 index: 16579 } } ignore_live_leader: false dest_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" is_pre_election: true
W20251026 03:11:39.145917 6861 leader_election.cc:343] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809): Illegal state: must be running to vote when last-logged opid is not known
I20251026 03:11:39.146509 6898 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "6db2d77e455540e793d960ebafe378df" candidate_term: 3 candidate_status { last_received { term: 2 index: 16579 } } ignore_live_leader: false dest_uuid: "1314cd2aaaf04cc48783327f6a3a5394" is_pre_election: true
I20251026 03:11:39.146575 6898 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db2d77e455540e793d960ebafe378df in term 2.
I20251026 03:11:39.146741 6972 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1314cd2aaaf04cc48783327f6a3a5394, 6db2d77e455540e793d960ebafe378df; no voters:
I20251026 03:11:39.146821 7152 raft_consensus.cc:2804] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 03:11:39.146871 7152 raft_consensus.cc:493] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Starting leader election (detected failure of leader d25b83ae8eaf44a6b917eff8b3e2ea86)
I20251026 03:11:39.146827 7010 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "1314cd2aaaf04cc48783327f6a3a5394" candidate_term: 3 candidate_status { last_received { term: 2 index: 16579 } } ignore_live_leader: false dest_uuid: "6db2d77e455540e793d960ebafe378df" is_pre_election: true
I20251026 03:11:39.146888 7118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "6db2d77e455540e793d960ebafe378df" candidate_term: 3 candidate_status { last_received { term: 2 index: 16579 } } ignore_live_leader: false dest_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" is_pre_election: true
I20251026 03:11:39.146891 7152 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 2 FOLLOWER]: Advancing to term 3
W20251026 03:11:39.147119 6973 leader_election.cc:343] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809): Illegal state: must be running to vote when last-logged opid is not known
I20251026 03:11:39.147778 7152 raft_consensus.cc:515] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:39.147909 7010 raft_consensus.cc:2393] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1314cd2aaaf04cc48783327f6a3a5394 in current term 3: Already voted for candidate 6db2d77e455540e793d960ebafe378df in this term.
I20251026 03:11:39.147928 7152 leader_election.cc:290] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 3 election: Requested vote from peers d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809), 1314cd2aaaf04cc48783327f6a3a5394 (127.30.128.65:38679)
I20251026 03:11:39.148048 7118 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "6db2d77e455540e793d960ebafe378df" candidate_term: 3 candidate_status { last_received { term: 2 index: 16579 } } ignore_live_leader: false dest_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86"
I20251026 03:11:39.148202 6898 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "feca937b535d4e818a8a8ffd9bb3096a" candidate_uuid: "6db2d77e455540e793d960ebafe378df" candidate_term: 3 candidate_status { last_received { term: 2 index: 16579 } } ignore_live_leader: false dest_uuid: "1314cd2aaaf04cc48783327f6a3a5394"
I20251026 03:11:39.148264 6898 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:11:39.148308 6860 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1314cd2aaaf04cc48783327f6a3a5394; no voters: 6db2d77e455540e793d960ebafe378df, d25b83ae8eaf44a6b917eff8b3e2ea86
W20251026 03:11:39.148267 6973 leader_election.cc:343] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809): Illegal state: must be running to vote when last-logged opid is not known
I20251026 03:11:39.148849 6898 raft_consensus.cc:2468] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6db2d77e455540e793d960ebafe378df in term 3.
I20251026 03:11:39.148979 7151 raft_consensus.cc:2749] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251026 03:11:39.148980 6972 leader_election.cc:304] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1314cd2aaaf04cc48783327f6a3a5394, 6db2d77e455540e793d960ebafe378df; no voters: d25b83ae8eaf44a6b917eff8b3e2ea86
I20251026 03:11:39.149056 7085 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap replayed 2/4 log segments. Stats: ops{read=9890 overwritten=0 applied=9888 ignored=0} inserts{seen=494350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:11:39.149103 7152 raft_consensus.cc:2804] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 3 FOLLOWER]: Leader election won for term 3
I20251026 03:11:39.149227 7152 raft_consensus.cc:697] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 3 LEADER]: Becoming Leader. State: Replica: 6db2d77e455540e793d960ebafe378df, State: Running, Role: LEADER
I20251026 03:11:39.149364 7152 consensus_queue.cc:237] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16579, Committed index: 16579, Last appended: 2.16579, Last appended by leader: 16579, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:39.149840 6488 catalog_manager.cc:5649] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df reported cstate change: term changed from 2 to 3, leader changed from d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67) to 6db2d77e455540e793d960ebafe378df (127.30.128.66). New cstate: current_term: 3 leader_uuid: "6db2d77e455540e793d960ebafe378df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } health_report { overall_health: UNKNOWN } } }
I20251026 03:11:39.655066 6898 raft_consensus.cc:1275] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 3 FOLLOWER]: Refusing update from remote peer 6db2d77e455540e793d960ebafe378df: Log matching property violated. Preceding OpId in replica: term: 2 index: 16579. Preceding OpId from leader: term: 3 index: 16580. (index mismatch)
I20251026 03:11:39.655333 7167 consensus_queue.cc:1048] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [LEADER]: Connected to new peer: Peer: permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16580, Last known committed idx: 16579, Time since last communication: 0.000s
W20251026 03:11:39.656605 6973 consensus_peers.cc:597] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df -> Peer d25b83ae8eaf44a6b917eff8b3e2ea86 (127.30.128.67:39809): Couldn't send request to peer d25b83ae8eaf44a6b917eff8b3e2ea86. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 03:11:39.939410 7085 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap replayed 3/4 log segments. Stats: ops{read=14843 overwritten=0 applied=14841 ignored=0} inserts{seen=742000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:11:40.065448 7085 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap replayed 4/4 log segments. Stats: ops{read=16579 overwritten=0 applied=16579 ignored=0} inserts{seen=765550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:11:40.065943 7085 tablet_bootstrap.cc:492] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Bootstrap complete.
I20251026 03:11:40.070865 7085 ts_tablet_manager.cc:1403] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Time spent bootstrapping tablet: real 2.390s user 2.032s sys 0.355s
I20251026 03:11:40.071012 7085 raft_consensus.cc:359] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:40.071072 7085 raft_consensus.cc:740] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d25b83ae8eaf44a6b917eff8b3e2ea86, State: Initialized, Role: FOLLOWER
I20251026 03:11:40.071111 7085 consensus_queue.cc:260] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16579, Last appended: 2.16579, Last appended by leader: 16579, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db2d77e455540e793d960ebafe378df" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 33983 } } peers { permanent_uuid: "d25b83ae8eaf44a6b917eff8b3e2ea86" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 39809 } } peers { permanent_uuid: "1314cd2aaaf04cc48783327f6a3a5394" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 38679 } }
I20251026 03:11:40.071288 7085 ts_tablet_manager.cc:1434] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20251026 03:11:40.104103 7108 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36874: Illegal state: replica d25b83ae8eaf44a6b917eff8b3e2ea86 is not leader of this config: current role FOLLOWER
I20251026 03:11:40.203950 7118 raft_consensus.cc:3060] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:11:40.970026 31233 tablet_server.cc:178] TabletServer@127.30.128.65:38679 shutting down...
I20251026 03:11:40.972306 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:40.972406 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:11:40.972460 31233 raft_consensus.cc:2243] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:40.972489 31233 raft_consensus.cc:2272] T feca937b535d4e818a8a8ffd9bb3096a P 1314cd2aaaf04cc48783327f6a3a5394 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:40.977753 31233 tablet_server.cc:195] TabletServer@127.30.128.65:38679 shutdown complete.
I20251026 03:11:40.978924 31233 tablet_server.cc:178] TabletServer@127.30.128.66:33983 shutting down...
I20251026 03:11:40.981401 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:40.981485 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:11:40.981539 31233 raft_consensus.cc:2243] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 3 LEADER]: Raft consensus shutting down.
I20251026 03:11:40.981590 31233 raft_consensus.cc:2272] T feca937b535d4e818a8a8ffd9bb3096a P 6db2d77e455540e793d960ebafe378df [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:41.002336 31233 tablet_server.cc:195] TabletServer@127.30.128.66:33983 shutdown complete.
I20251026 03:11:41.003350 31233 tablet_server.cc:178] TabletServer@127.30.128.67:39809 shutting down...
I20251026 03:11:41.005546 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:41.005650 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:11:41.005697 31233 raft_consensus.cc:2243] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:41.005741 31233 raft_consensus.cc:2272] T feca937b535d4e818a8a8ffd9bb3096a P d25b83ae8eaf44a6b917eff8b3e2ea86 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:41.020206 31233 tablet_server.cc:195] TabletServer@127.30.128.67:39809 shutdown complete.
I20251026 03:11:41.021199 31233 master.cc:561] Master@127.30.128.126:37409 shutting down...
I20251026 03:11:41.023331 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:11:41.023386 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:41.023402 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4ce72fd7240e4924b43c91f8d2ec9f77: stopping tablet replica
I20251026 03:11:41.034751 31233 master.cc:583] Master@127.30.128.126:37409 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientSucceedsOnBootstrap (19676 ms)
[ RUN ] TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound
I20251026 03:11:41.054800 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:35813
I20251026 03:11:41.054929 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:41.056032 7187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.056195 7190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.056165 7188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:41.056083 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:41.056387 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:41.056437 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:41.056450 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448301056451 us; error 0 us; skew 500 ppm
I20251026 03:11:41.057011 31233 webserver.cc:492] Webserver started at http://127.30.128.126:34923/ using document root <none> and password file <none>
I20251026 03:11:41.057072 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:41.057102 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:41.057133 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:41.057338 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "0c7d734b6a184881b8ca2b1e1899fa8b"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.058123 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:41.058475 7195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.058573 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.058612 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "0c7d734b6a184881b8ca2b1e1899fa8b"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.058650 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:41.064260 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:41.064399 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:41.066733 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:35813
I20251026 03:11:41.069409 7247 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:35813 every 8 connection(s)
I20251026 03:11:41.069520 7248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:41.070384 7248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b: Bootstrap starting.
I20251026 03:11:41.070596 7248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:41.070945 7248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b: No bootstrap required, opened a new log
I20251026 03:11:41.071036 7248 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" member_type: VOTER }
I20251026 03:11:41.071072 7248 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:41.071084 7248 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7d734b6a184881b8ca2b1e1899fa8b, State: Initialized, Role: FOLLOWER
I20251026 03:11:41.071116 7248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" member_type: VOTER }
I20251026 03:11:41.071158 7248 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:11:41.071177 7248 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:11:41.071194 7248 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:41.071605 7248 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" member_type: VOTER }
I20251026 03:11:41.071656 7248 leader_election.cc:304] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7d734b6a184881b8ca2b1e1899fa8b; no voters:
I20251026 03:11:41.071745 7248 leader_election.cc:290] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:11:41.071781 7251 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:41.071868 7248 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:11:41.071890 7251 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 1 LEADER]: Becoming Leader. State: Replica: 0c7d734b6a184881b8ca2b1e1899fa8b, State: Running, Role: LEADER
I20251026 03:11:41.072027 7251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" member_type: VOTER }
I20251026 03:11:41.072214 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7d734b6a184881b8ca2b1e1899fa8b. Latest consensus state: current_term: 1 leader_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" member_type: VOTER } }
I20251026 03:11:41.072317 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:41.072211 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7d734b6a184881b8ca2b1e1899fa8b" member_type: VOTER } }
I20251026 03:11:41.072464 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:41.072634 7257 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:11:41.072774 7257 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:11:41.073235 7257 catalog_manager.cc:1357] Generated new cluster ID: d378eaff64aa4e679ccfef07c2c9d8e5
I20251026 03:11:41.073275 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:11:41.073282 7257 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:11:41.078576 7257 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:11:41.078926 7257 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:11:41.082252 7257 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b: Generated new TSK 0
I20251026 03:11:41.082304 7257 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:11:41.088837 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:41.089902 7270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.089926 7269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:41.090039 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:41.089927 7272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:41.090209 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:41.090242 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:41.090263 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448301090263 us; error 0 us; skew 500 ppm
I20251026 03:11:41.090857 31233 webserver.cc:492] Webserver started at http://127.30.128.65:45927/ using document root <none> and password file <none>
I20251026 03:11:41.090937 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:41.090983 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:41.091032 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:41.091264 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "818b0f6aa49b49e2bf960a25a316146f"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.092049 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:41.092456 7277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.092568 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.092607 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "818b0f6aa49b49e2bf960a25a316146f"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.092651 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:41.099965 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:41.100112 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:41.100219 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:41.100370 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:41.100394 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.100417 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:41.100440 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.102870 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:43911
I20251026 03:11:41.103410 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:11:41.104765 7340 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:43911 every 8 connection(s)
W20251026 03:11:41.105521 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.105543 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.105789 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:41.106109 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:41.106230 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:41.106262 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:41.106284 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448301106284 us; error 0 us; skew 500 ppm
I20251026 03:11:41.106887 31233 webserver.cc:492] Webserver started at http://127.30.128.66:44981/ using document root <none> and password file <none>
I20251026 03:11:41.106961 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:41.106995 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:41.107043 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:41.107280 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "0c9b28898180464cab5f5ddc21ceba1a"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.108191 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:41.108209 7341 heartbeater.cc:344] Connected to a master server at 127.30.128.126:35813
I20251026 03:11:41.108256 7341 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:41.108350 7341 heartbeater.cc:507] Master 127.30.128.126:35813 requested a full tablet report, sending...
I20251026 03:11:41.108536 7212 ts_manager.cc:194] Registered new tserver with Master: 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65:43911)
I20251026 03:11:41.108651 7353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.108755 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.108798 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "0c9b28898180464cab5f5ddc21ceba1a"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.108839 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:41.109304 7212 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59852
I20251026 03:11:41.113888 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:41.114018 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:41.114122 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:41.114254 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:41.114275 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.114298 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:41.114315 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.116853 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:32921
I20251026 03:11:41.116905 7416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:32921 every 8 connection(s)
I20251026 03:11:41.117162 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:41.118090 7420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.118149 7421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:41.118512 7423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:41.118741 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:41.118861 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:41.118887 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:41.118906 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448301118906 us; error 0 us; skew 500 ppm
I20251026 03:11:41.119472 31233 webserver.cc:492] Webserver started at http://127.30.128.67:36101/ using document root <none> and password file <none>
I20251026 03:11:41.119554 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:41.119596 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:41.119690 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:41.119994 7417 heartbeater.cc:344] Connected to a master server at 127.30.128.126:35813
I20251026 03:11:41.120059 7417 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:41.120018 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.120143 7417 heartbeater.cc:507] Master 127.30.128.126:35813 requested a full tablet report, sending...
I20251026 03:11:41.120348 7212 ts_manager.cc:194] Registered new tserver with Master: 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66:32921)
I20251026 03:11:41.120782 7212 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59854
I20251026 03:11:41.121011 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:41.121430 7428 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.121512 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.121546 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
format_stamp: "Formatted at 2025-10-26 03:11:41 on dist-test-slave-51kf"
I20251026 03:11:41.121575 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:41.127305 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:41.127442 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:41.127537 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:41.127658 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:41.127678 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.127696 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:41.127712 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.130307 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:45319
I20251026 03:11:41.130353 7491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:45319 every 8 connection(s)
I20251026 03:11:41.132819 7492 heartbeater.cc:344] Connected to a master server at 127.30.128.126:35813
I20251026 03:11:41.132865 7492 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:41.132927 7492 heartbeater.cc:507] Master 127.30.128.126:35813 requested a full tablet report, sending...
I20251026 03:11:41.133114 7212 ts_manager.cc:194] Registered new tserver with Master: 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319)
I20251026 03:11:41.133447 7212 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59862
I20251026 03:11:41.133740 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00328704s
I20251026 03:11:41.134040 31233 test_util.cc:276] Using random seed: 608665816
I20251026 03:11:41.137943 7212 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59872:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:11:41.138063 7212 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:11:41.141981 7456 tablet_service.cc:1505] Processing CreateTablet for tablet fdd186a3efbf49f6bb503bc0de3795ca (DEFAULT_TABLE table=test-workload [id=f54578ce51624c499d8075ff176628e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:41.142009 7381 tablet_service.cc:1505] Processing CreateTablet for tablet fdd186a3efbf49f6bb503bc0de3795ca (DEFAULT_TABLE table=test-workload [id=f54578ce51624c499d8075ff176628e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:41.142123 7381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd186a3efbf49f6bb503bc0de3795ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:41.142568 7305 tablet_service.cc:1505] Processing CreateTablet for tablet fdd186a3efbf49f6bb503bc0de3795ca (DEFAULT_TABLE table=test-workload [id=f54578ce51624c499d8075ff176628e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:41.142665 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd186a3efbf49f6bb503bc0de3795ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:41.142124 7456 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd186a3efbf49f6bb503bc0de3795ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:41.143422 7507 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: Bootstrap starting.
I20251026 03:11:41.143905 7508 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap starting.
I20251026 03:11:41.143841 7507 tablet_bootstrap.cc:654] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:41.144309 7508 tablet_bootstrap.cc:654] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:41.144500 7510 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2: Bootstrap starting.
I20251026 03:11:41.144563 7507 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: No bootstrap required, opened a new log
I20251026 03:11:41.144613 7507 ts_tablet_manager.cc:1403] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:41.144850 7507 raft_consensus.cc:359] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.144913 7510 tablet_bootstrap.cc:654] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:41.144938 7507 raft_consensus.cc:385] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:41.144974 7507 raft_consensus.cc:740] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9b28898180464cab5f5ddc21ceba1a, State: Initialized, Role: FOLLOWER
I20251026 03:11:41.145042 7507 consensus_queue.cc:260] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.145175 7507 ts_tablet_manager.cc:1434] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:41.145210 7417 heartbeater.cc:499] Master 127.30.128.126:35813 was elected leader, sending a full tablet report...
I20251026 03:11:41.145545 7508 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: No bootstrap required, opened a new log
I20251026 03:11:41.145550 7510 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2: No bootstrap required, opened a new log
I20251026 03:11:41.145620 7508 ts_tablet_manager.cc:1403] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:11:41.145625 7510 ts_tablet_manager.cc:1403] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:41.145808 7510 raft_consensus.cc:359] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.145808 7508 raft_consensus.cc:359] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.145900 7510 raft_consensus.cc:385] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:41.145910 7508 raft_consensus.cc:385] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:41.145979 7510 raft_consensus.cc:740] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be3b14b61ec4d0fb554bf9279d98ad2, State: Initialized, Role: FOLLOWER
I20251026 03:11:41.145996 7508 raft_consensus.cc:740] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b0f6aa49b49e2bf960a25a316146f, State: Initialized, Role: FOLLOWER
I20251026 03:11:41.146046 7508 consensus_queue.cc:260] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.146047 7510 consensus_queue.cc:260] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.146232 7508 ts_tablet_manager.cc:1434] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:41.146238 7510 ts_tablet_manager.cc:1434] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 03:11:41.146289 7341 heartbeater.cc:499] Master 127.30.128.126:35813 was elected leader, sending a full tablet report...
I20251026 03:11:41.146421 7492 heartbeater.cc:499] Master 127.30.128.126:35813 was elected leader, sending a full tablet report...
I20251026 03:11:41.162171 7512 raft_consensus.cc:493] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:11:41.162240 7512 raft_consensus.cc:515] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.162391 7512 leader_election.cc:290] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319), 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65:43911)
I20251026 03:11:41.165407 7466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" is_pre_election: true
I20251026 03:11:41.165409 7315 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b0f6aa49b49e2bf960a25a316146f" is_pre_election: true
I20251026 03:11:41.165527 7466 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9b28898180464cab5f5ddc21ceba1a in term 0.
I20251026 03:11:41.165537 7315 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9b28898180464cab5f5ddc21ceba1a in term 0.
I20251026 03:11:41.165699 7354 leader_election.cc:304] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9b28898180464cab5f5ddc21ceba1a, 5be3b14b61ec4d0fb554bf9279d98ad2; no voters:
I20251026 03:11:41.165789 7512 raft_consensus.cc:2804] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:11:41.165833 7512 raft_consensus.cc:493] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:11:41.165858 7512 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:41.166527 7512 raft_consensus.cc:515] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.166638 7512 leader_election.cc:290] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 1 election: Requested vote from peers 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319), 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65:43911)
I20251026 03:11:41.166738 7315 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b0f6aa49b49e2bf960a25a316146f"
I20251026 03:11:41.166801 7315 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:41.166756 7466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
I20251026 03:11:41.166850 7466 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:41.167354 7315 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9b28898180464cab5f5ddc21ceba1a in term 1.
I20251026 03:11:41.167403 7466 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9b28898180464cab5f5ddc21ceba1a in term 1.
I20251026 03:11:41.167498 7355 leader_election.cc:304] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9b28898180464cab5f5ddc21ceba1a, 818b0f6aa49b49e2bf960a25a316146f; no voters:
I20251026 03:11:41.167593 7512 raft_consensus.cc:2804] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:41.167704 7512 raft_consensus.cc:697] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 LEADER]: Becoming Leader. State: Replica: 0c9b28898180464cab5f5ddc21ceba1a, State: Running, Role: LEADER
I20251026 03:11:41.167758 7512 consensus_queue.cc:237] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:41.168198 7212 catalog_manager.cc:5649] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66). New cstate: current_term: 1 leader_uuid: "0c9b28898180464cab5f5ddc21ceba1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } health_report { overall_health: HEALTHY } } }
I20251026 03:11:41.184149 7315 raft_consensus.cc:1275] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 1 FOLLOWER]: Refusing update from remote peer 0c9b28898180464cab5f5ddc21ceba1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:41.184149 7466 raft_consensus.cc:1275] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 1 FOLLOWER]: Refusing update from remote peer 0c9b28898180464cab5f5ddc21ceba1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:41.184474 7517 consensus_queue.cc:1048] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:41.184585 7512 consensus_queue.cc:1048] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:51.702092 7529 raft_consensus.cc:993] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: : Instructing follower 818b0f6aa49b49e2bf960a25a316146f to start an election
I20251026 03:11:51.702169 7529 raft_consensus.cc:1081] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 LEADER]: Signalling peer 818b0f6aa49b49e2bf960a25a316146f to start an election
I20251026 03:11:51.702366 7315 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca"
dest_uuid: "818b0f6aa49b49e2bf960a25a316146f"
from {username='slave'} at 127.0.0.1:57096
I20251026 03:11:51.702432 7315 raft_consensus.cc:493] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:11:51.702451 7315 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:51.703047 7315 raft_consensus.cc:515] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:51.703183 7315 leader_election.cc:290] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [CANDIDATE]: Term 2 election: Requested vote from peers 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319), 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66:32921)
I20251026 03:11:51.705788 7391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "818b0f6aa49b49e2bf960a25a316146f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14933 } } ignore_live_leader: true dest_uuid: "0c9b28898180464cab5f5ddc21ceba1a"
I20251026 03:11:51.705765 7466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "818b0f6aa49b49e2bf960a25a316146f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14933 } } ignore_live_leader: true dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
I20251026 03:11:51.705876 7391 raft_consensus.cc:3055] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:11:51.705904 7466 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:51.705905 7391 raft_consensus.cc:740] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c9b28898180464cab5f5ddc21ceba1a, State: Running, Role: LEADER
I20251026 03:11:51.706005 7391 consensus_queue.cc:260] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14933, Committed index: 14933, Last appended: 1.14933, Last appended by leader: 14933, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:51.706096 7391 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:11:51.706557 7466 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b0f6aa49b49e2bf960a25a316146f in term 2.
I20251026 03:11:51.706707 7391 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b0f6aa49b49e2bf960a25a316146f in term 2.
I20251026 03:11:51.706753 7278 leader_election.cc:304] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5be3b14b61ec4d0fb554bf9279d98ad2, 818b0f6aa49b49e2bf960a25a316146f; no voters:
I20251026 03:11:51.706914 7572 raft_consensus.cc:2804] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:11:51.707046 7572 raft_consensus.cc:697] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 LEADER]: Becoming Leader. State: Replica: 818b0f6aa49b49e2bf960a25a316146f, State: Running, Role: LEADER
I20251026 03:11:51.707108 7572 consensus_queue.cc:237] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14933, Committed index: 14933, Last appended: 1.14933, Last appended by leader: 14933, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:51.707581 7211 catalog_manager.cc:5649] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f reported cstate change: term changed from 1 to 2, leader changed from 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66) to 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65). New cstate: current_term: 2 leader_uuid: "818b0f6aa49b49e2bf960a25a316146f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } health_report { overall_health: UNKNOWN } } }
I20251026 03:11:51.715170 31233 tablet_replica.cc:333] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: stopping tablet replica
I20251026 03:11:51.715215 31233 raft_consensus.cc:2243] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:11:51.715256 31233 pending_rounds.cc:70] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Trying to abort 1 pending ops.
I20251026 03:11:51.715281 31233 pending_rounds.cc:77] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Aborting op as it isn't in flight: id { term: 2 index: 14934 } timestamp: 7214892284752785408 op_type: NO_OP noop_request { }
I20251026 03:11:51.715318 31233 raft_consensus.cc:2889] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:11:51.715349 31233 raft_consensus.cc:2272] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:51.723281 31233 ts_tablet_manager.cc:1916] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 03:11:51.724534 31233 ts_tablet_manager.cc:1929] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.14934
I20251026 03:11:51.724581 31233 log.cc:1199] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Deleting WAL directory at /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/fdd186a3efbf49f6bb503bc0de3795ca
I20251026 03:11:53.680245 7587 raft_consensus.cc:493] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:11:53.680349 7587 raft_consensus.cc:515] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:53.680503 7587 leader_election.cc:290] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319), 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65:43911)
I20251026 03:11:53.680717 7315 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 14933 } } ignore_live_leader: false dest_uuid: "818b0f6aa49b49e2bf960a25a316146f" is_pre_election: true
I20251026 03:11:53.680722 7466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 14933 } } ignore_live_leader: false dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" is_pre_election: true
I20251026 03:11:53.680783 7315 raft_consensus.cc:1803] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.14934
I20251026 03:11:53.680868 7466 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9b28898180464cab5f5ddc21ceba1a in term 2.
I20251026 03:11:53.680893 7315 raft_consensus.cc:2410] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c9b28898180464cab5f5ddc21ceba1a for term 3 because replica has last-logged OpId of term: 2 index: 14934, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 14933.
I20251026 03:11:53.681021 7354 leader_election.cc:304] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9b28898180464cab5f5ddc21ceba1a, 5be3b14b61ec4d0fb554bf9279d98ad2; no voters:
I20251026 03:11:53.681119 7587 raft_consensus.cc:2804] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 03:11:53.681162 7587 raft_consensus.cc:493] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:11:53.681187 7587 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:11:53.681948 7587 raft_consensus.cc:515] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:53.682078 7587 leader_election.cc:290] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 3 election: Requested vote from peers 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319), 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65:43911)
I20251026 03:11:53.682193 7466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 14933 } } ignore_live_leader: false dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
I20251026 03:11:53.682235 7466 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:11:53.682219 7315 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "0c9b28898180464cab5f5ddc21ceba1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 14933 } } ignore_live_leader: false dest_uuid: "818b0f6aa49b49e2bf960a25a316146f"
I20251026 03:11:53.682319 7315 raft_consensus.cc:1803] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 2.14934
I20251026 03:11:53.682353 7315 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:11:53.682813 7466 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9b28898180464cab5f5ddc21ceba1a in term 3.
I20251026 03:11:53.682976 7354 leader_election.cc:304] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9b28898180464cab5f5ddc21ceba1a, 5be3b14b61ec4d0fb554bf9279d98ad2; no voters:
I20251026 03:11:53.682984 7315 raft_consensus.cc:2410] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c9b28898180464cab5f5ddc21ceba1a for term 3 because replica has last-logged OpId of term: 2 index: 14934, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 14933.
I20251026 03:11:53.683060 7587 raft_consensus.cc:2804] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 FOLLOWER]: Leader election won for term 3
I20251026 03:11:53.683162 7587 raft_consensus.cc:697] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 LEADER]: Becoming Leader. State: Replica: 0c9b28898180464cab5f5ddc21ceba1a, State: Running, Role: LEADER
I20251026 03:11:53.683262 7587 consensus_queue.cc:237] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14933, Committed index: 14933, Last appended: 1.14933, Last appended by leader: 14933, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:53.683792 7211 catalog_manager.cc:5649] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a reported cstate change: term changed from 2 to 3, leader changed from 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65) to 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66). New cstate: current_term: 3 leader_uuid: "0c9b28898180464cab5f5ddc21ceba1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } health_report { overall_health: HEALTHY } } }
I20251026 03:11:53.719545 7466 raft_consensus.cc:1275] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 3 FOLLOWER]: Refusing update from remote peer 0c9b28898180464cab5f5ddc21ceba1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 14933. Preceding OpId from leader: term: 3 index: 14935. (index mismatch)
W20251026 03:11:53.719632 7355 consensus_peers.cc:597] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a -> Peer 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65:43911): Couldn't send request to peer 818b0f6aa49b49e2bf960a25a316146f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 03:11:53.719776 7588 consensus_queue.cc:1048] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14934, Last known committed idx: 14933, Time since last communication: 0.000s
I20251026 03:11:54.190095 7604 ts_tablet_manager.cc:933] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Initiating tablet copy from peer 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66:32921)
I20251026 03:11:54.190285 7604 tablet_copy_client.cc:323] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: tablet copy: Beginning tablet copy session from remote peer at address 127.30.128.66:32921
I20251026 03:11:54.190575 7401 tablet_copy_service.cc:140] P 0c9b28898180464cab5f5ddc21ceba1a: Received BeginTabletCopySession request for tablet fdd186a3efbf49f6bb503bc0de3795ca from peer 818b0f6aa49b49e2bf960a25a316146f ({username='slave'} at 127.0.0.1:43038)
I20251026 03:11:54.190634 7401 tablet_copy_service.cc:161] P 0c9b28898180464cab5f5ddc21ceba1a: Beginning new tablet copy session on tablet fdd186a3efbf49f6bb503bc0de3795ca from peer 818b0f6aa49b49e2bf960a25a316146f at {username='slave'} at 127.0.0.1:43038: session id = 818b0f6aa49b49e2bf960a25a316146f-fdd186a3efbf49f6bb503bc0de3795ca
I20251026 03:11:54.191144 7401 tablet_copy_source_session.cc:215] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: Tablet Copy: opened 0 blocks and 4 log segments
I20251026 03:11:54.191514 7604 ts_tablet_manager.cc:1916] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Deleting tablet data with delete state TABLET_DATA_COPYING
I20251026 03:11:54.193141 7604 ts_tablet_manager.cc:1929] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.14934
I20251026 03:11:54.193231 7604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd186a3efbf49f6bb503bc0de3795ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:54.193723 7604 tablet_copy_client.cc:806] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: tablet copy: Starting download of 0 data blocks...
I20251026 03:11:54.193867 7604 tablet_copy_client.cc:670] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: tablet copy: Starting download of 4 WAL segments...
I20251026 03:11:54.230114 7604 tablet_copy_client.cc:538] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 03:11:54.230861 7604 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap starting.
I20251026 03:11:54.913976 7604 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap replayed 1/4 log segments. Stats: ops{read=4622 overwritten=0 applied=4620 ignored=0} inserts{seen=230950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:11:55.643105 7604 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap replayed 2/4 log segments. Stats: ops{read=9244 overwritten=0 applied=9241 ignored=0} inserts{seen=462000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:11:56.351824 7604 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap replayed 3/4 log segments. Stats: ops{read=13864 overwritten=0 applied=13864 ignored=0} inserts{seen=693150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:11:56.530499 7604 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap replayed 4/4 log segments. Stats: ops{read=14998 overwritten=0 applied=14997 ignored=0} inserts{seen=746600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:11:56.530748 7604 tablet_bootstrap.cc:492] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Bootstrap complete.
I20251026 03:11:56.535526 7604 ts_tablet_manager.cc:1403] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Time spent bootstrapping tablet: real 2.305s user 1.979s sys 0.324s
I20251026 03:11:56.535686 7604 raft_consensus.cc:359] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:56.535847 7604 raft_consensus.cc:740] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b0f6aa49b49e2bf960a25a316146f, State: Initialized, Role: FOLLOWER
I20251026 03:11:56.535902 7604 consensus_queue.cc:260] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14997, Last appended: 3.14998, Last appended by leader: 14998, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:56.536062 7604 ts_tablet_manager.cc:1434] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20251026 03:11:56.536254 7401 tablet_copy_service.cc:342] P 0c9b28898180464cab5f5ddc21ceba1a: Request end of tablet copy session 818b0f6aa49b49e2bf960a25a316146f-fdd186a3efbf49f6bb503bc0de3795ca received from {username='slave'} at 127.0.0.1:43038
I20251026 03:11:56.536319 7401 tablet_copy_service.cc:434] P 0c9b28898180464cab5f5ddc21ceba1a: ending tablet copy session 818b0f6aa49b49e2bf960a25a316146f-fdd186a3efbf49f6bb503bc0de3795ca on tablet fdd186a3efbf49f6bb503bc0de3795ca with peer 818b0f6aa49b49e2bf960a25a316146f
I20251026 03:11:56.622054 7315 raft_consensus.cc:1217] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.14933->[3.14934-3.15202] Dedup: 3.14998->[3.14999-3.15202]
I20251026 03:11:56.627460 7616 raft_consensus.cc:993] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: : Instructing follower 5be3b14b61ec4d0fb554bf9279d98ad2 to start an election
I20251026 03:11:56.627544 7616 raft_consensus.cc:1081] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 LEADER]: Signalling peer 5be3b14b61ec4d0fb554bf9279d98ad2 to start an election
I20251026 03:11:56.627552 7587 raft_consensus.cc:993] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: : Instructing follower 818b0f6aa49b49e2bf960a25a316146f to start an election
I20251026 03:11:56.627743 7587 raft_consensus.cc:1081] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 LEADER]: Signalling peer 818b0f6aa49b49e2bf960a25a316146f to start an election
I20251026 03:11:56.627943 7466 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca"
dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
from {username='slave'} at 127.0.0.1:41580
W20251026 03:11:56.628132 7354 consensus_peers.cc:409] unable to start election on peer 5be3b14b61ec4d0fb554bf9279d98ad2: Illegal state: leader elections are disabled
I20251026 03:11:56.628190 7315 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca"
dest_uuid: "818b0f6aa49b49e2bf960a25a316146f"
from {username='slave'} at 127.0.0.1:57096
I20251026 03:11:56.628234 7315 raft_consensus.cc:493] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 03:11:56.628257 7315 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:11:56.629027 7315 raft_consensus.cc:515] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:56.629164 7315 leader_election.cc:290] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [CANDIDATE]: Term 4 election: Requested vote from peers 5be3b14b61ec4d0fb554bf9279d98ad2 (127.30.128.67:45319), 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66:32921)
I20251026 03:11:56.629307 7391 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "818b0f6aa49b49e2bf960a25a316146f" candidate_term: 4 candidate_status { last_received { term: 3 index: 15204 } } ignore_live_leader: true dest_uuid: "0c9b28898180464cab5f5ddc21ceba1a"
I20251026 03:11:56.629382 7391 raft_consensus.cc:3055] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 LEADER]: Stepping down as leader of term 3
I20251026 03:11:56.629395 7466 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fdd186a3efbf49f6bb503bc0de3795ca" candidate_uuid: "818b0f6aa49b49e2bf960a25a316146f" candidate_term: 4 candidate_status { last_received { term: 3 index: 15204 } } ignore_live_leader: true dest_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2"
I20251026 03:11:56.629401 7391 raft_consensus.cc:740] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0c9b28898180464cab5f5ddc21ceba1a, State: Running, Role: LEADER
I20251026 03:11:56.629459 7466 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:11:56.629519 7391 consensus_queue.cc:260] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15204, Committed index: 15204, Last appended: 3.15204, Last appended by leader: 15204, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:56.629617 7391 raft_consensus.cc:3060] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:11:56.630070 7466 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b0f6aa49b49e2bf960a25a316146f in term 4.
I20251026 03:11:56.630201 7278 leader_election.cc:304] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5be3b14b61ec4d0fb554bf9279d98ad2, 818b0f6aa49b49e2bf960a25a316146f; no voters:
I20251026 03:11:56.630286 7629 raft_consensus.cc:2804] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 4 FOLLOWER]: Leader election won for term 4
I20251026 03:11:56.630358 7391 raft_consensus.cc:2468] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 818b0f6aa49b49e2bf960a25a316146f in term 4.
I20251026 03:11:56.630370 7629 raft_consensus.cc:697] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 4 LEADER]: Becoming Leader. State: Replica: 818b0f6aa49b49e2bf960a25a316146f, State: Running, Role: LEADER
I20251026 03:11:56.630420 7629 consensus_queue.cc:237] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15204, Committed index: 15204, Last appended: 3.15204, Last appended by leader: 15204, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } }
I20251026 03:11:56.630709 7211 catalog_manager.cc:5649] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f reported cstate change: term changed from 3 to 4, leader changed from 0c9b28898180464cab5f5ddc21ceba1a (127.30.128.66) to 818b0f6aa49b49e2bf960a25a316146f (127.30.128.65). New cstate: current_term: 4 leader_uuid: "818b0f6aa49b49e2bf960a25a316146f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5be3b14b61ec4d0fb554bf9279d98ad2" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 45319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "818b0f6aa49b49e2bf960a25a316146f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9b28898180464cab5f5ddc21ceba1a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32921 } health_report { overall_health: UNKNOWN } } }
I20251026 03:11:56.645320 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:11:56.647511 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:56.647657 31233 tablet_replica.cc:333] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: stopping tablet replica
I20251026 03:11:56.647739 31233 raft_consensus.cc:2243] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 4 LEADER]: Raft consensus shutting down.
I20251026 03:11:56.647895 31233 pending_rounds.cc:70] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Trying to abort 1 pending ops.
I20251026 03:11:56.647919 31233 pending_rounds.cc:77] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f: Aborting op as it isn't in flight: id { term: 4 index: 15205 } timestamp: 7214892304918618112 op_type: NO_OP noop_request { }
I20251026 03:11:56.647981 31233 raft_consensus.cc:2889] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 4 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:11:56.648011 31233 raft_consensus.cc:2272] T fdd186a3efbf49f6bb503bc0de3795ca P 818b0f6aa49b49e2bf960a25a316146f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:56.661536 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:11:56.662459 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:11:56.664743 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:56.664842 31233 tablet_replica.cc:333] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a: stopping tablet replica
I20251026 03:11:56.664912 31233 raft_consensus.cc:2243] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:56.664968 31233 raft_consensus.cc:2272] T fdd186a3efbf49f6bb503bc0de3795ca P 0c9b28898180464cab5f5ddc21ceba1a [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:56.688110 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:11:56.689046 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:11:56.691264 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:11:56.691357 31233 tablet_replica.cc:333] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2: stopping tablet replica
I20251026 03:11:56.691392 31233 raft_consensus.cc:2243] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251026 03:11:56.691439 31233 raft_consensus.cc:2272] T fdd186a3efbf49f6bb503bc0de3795ca P 5be3b14b61ec4d0fb554bf9279d98ad2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:56.694736 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:11:56.695626 31233 master.cc:561] Master@127.30.128.126:35813 shutting down...
I20251026 03:11:56.697499 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:11:56.697557 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:11:56.697590 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0c7d734b6a184881b8ca2b1e1899fa8b: stopping tablet replica
I20251026 03:11:56.709203 31233 master.cc:583] Master@127.30.128.126:35813 shutdown complete.
[ OK ] TxnParticipantITest.TestTxnSystemClientRetriesWhenReplicaNotFound (15671 ms)
[----------] 21 tests from TxnParticipantITest (251863 ms total)
[----------] 2 tests from TxnParticipantElectionStormITest
[ RUN ] TxnParticipantElectionStormITest.TestFrequentElections
I20251026 03:11:56.726390 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:45221
I20251026 03:11:56.726508 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:56.727972 7639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.728019 7641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:56.728142 31233 server_base.cc:1047] running on GCE node
W20251026 03:11:56.728179 7638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:56.728327 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:56.728366 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:56.728384 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448316728384 us; error 0 us; skew 500 ppm
I20251026 03:11:56.728874 31233 webserver.cc:492] Webserver started at http://127.30.128.126:43377/ using document root <none> and password file <none>
I20251026 03:11:56.728943 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:56.728984 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:56.729038 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:56.729259 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "46176127b3ae434bbeba5dfb6dd21469"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.730047 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:56.730412 7646 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.730535 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.730574 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "46176127b3ae434bbeba5dfb6dd21469"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.730624 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:56.739007 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:56.739151 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:56.741398 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:45221
I20251026 03:11:56.743952 7698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:45221 every 8 connection(s)
I20251026 03:11:56.751346 7699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:56.752228 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Bootstrap starting.
I20251026 03:11:56.752477 7699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:56.752907 7699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: No bootstrap required, opened a new log
I20251026 03:11:56.753013 7699 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:11:56.753060 7699 raft_consensus.cc:385] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:56.753073 7699 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46176127b3ae434bbeba5dfb6dd21469, State: Initialized, Role: FOLLOWER
I20251026 03:11:56.753119 7699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:11:56.753154 7699 raft_consensus.cc:399] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:11:56.753180 7699 raft_consensus.cc:493] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:11:56.753206 7699 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:56.753623 7699 raft_consensus.cc:515] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:11:56.753671 7699 leader_election.cc:304] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46176127b3ae434bbeba5dfb6dd21469; no voters:
I20251026 03:11:56.753798 7699 leader_election.cc:290] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:11:56.753846 7702 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:56.753937 7699 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:11:56.753958 7702 raft_consensus.cc:697] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 LEADER]: Becoming Leader. State: Replica: 46176127b3ae434bbeba5dfb6dd21469, State: Running, Role: LEADER
I20251026 03:11:56.754087 7702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:11:56.754272 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46176127b3ae434bbeba5dfb6dd21469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER } }
I20251026 03:11:56.754280 7704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46176127b3ae434bbeba5dfb6dd21469. Latest consensus state: current_term: 1 leader_uuid: "46176127b3ae434bbeba5dfb6dd21469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER } }
I20251026 03:11:56.754374 7704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:56.754606 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: This master's current role is: LEADER
I20251026 03:11:56.754696 7708 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:11:56.754802 7708 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:11:56.755133 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:11:56.755323 7708 catalog_manager.cc:1357] Generated new cluster ID: 8d4f5ea956f944a4be2d0daa0829611b
I20251026 03:11:56.755365 7708 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:11:56.765714 7708 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:11:56.766112 7708 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:11:56.770033 7708 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Generated new TSK 0
I20251026 03:11:56.770098 7708 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:11:56.770678 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:56.771682 7720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.771726 7723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.771749 7721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:56.771829 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:56.771960 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:56.771987 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:56.772006 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448316772006 us; error 0 us; skew 500 ppm
I20251026 03:11:56.772500 31233 webserver.cc:492] Webserver started at http://127.30.128.65:45061/ using document root <none> and password file <none>
I20251026 03:11:56.772578 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:56.772617 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:56.772667 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:56.772907 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.773689 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:56.774158 7728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.774276 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.774317 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.774372 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:56.797166 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:56.797364 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:56.797523 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:56.797716 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:56.797744 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.797807 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:56.797840 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.800444 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:46839
I20251026 03:11:56.801251 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:11:56.802416 7791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:46839 every 8 connection(s)
W20251026 03:11:56.803128 7795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.803112 7794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.803294 7797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:56.803596 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:56.803682 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:56.803725 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:56.803750 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448316803750 us; error 0 us; skew 500 ppm
I20251026 03:11:56.804386 31233 webserver.cc:492] Webserver started at http://127.30.128.66:33749/ using document root <none> and password file <none>
I20251026 03:11:56.804471 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:56.804518 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:56.804567 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:56.804809 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "5340c78feac8406882e860a2bed2ec2a"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.805722 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:56.806202 7804 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.806326 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.806375 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "5340c78feac8406882e860a2bed2ec2a"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.806422 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:56.809718 7792 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45221
I20251026 03:11:56.809810 7792 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:56.809916 7792 heartbeater.cc:507] Master 127.30.128.126:45221 requested a full tablet report, sending...
I20251026 03:11:56.810163 7663 ts_manager.cc:194] Registered new tserver with Master: 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:11:56.810664 7663 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33060
I20251026 03:11:56.821080 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:56.821270 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:56.821388 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:56.821549 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:56.821573 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.821604 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:56.821626 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.824308 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:32837
I20251026 03:11:56.824375 7867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:32837 every 8 connection(s)
I20251026 03:11:56.824651 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:11:56.825598 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.825627 7874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:11:56.826021 7872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:11:56.826272 31233 server_base.cc:1047] running on GCE node
I20251026 03:11:56.826385 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:11:56.826423 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:11:56.826452 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448316826452 us; error 0 us; skew 500 ppm
I20251026 03:11:56.827167 31233 webserver.cc:492] Webserver started at http://127.30.128.67:34413/ using document root <none> and password file <none>
I20251026 03:11:56.827261 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:11:56.827311 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:11:56.827360 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:11:56.827430 7868 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45221
I20251026 03:11:56.827483 7868 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:56.827587 7868 heartbeater.cc:507] Master 127.30.128.126:45221 requested a full tablet report, sending...
I20251026 03:11:56.827659 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "360eb6050b7244d58989794aea84cc01"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.827803 7663 ts_manager.cc:194] Registered new tserver with Master: 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:11:56.828145 7663 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33068
I20251026 03:11:56.828613 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:56.829038 7879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.829166 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.829205 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "360eb6050b7244d58989794aea84cc01"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:11:56.829262 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:11:56.832026 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:11:56.832161 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:11:56.832274 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:11:56.832420 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:11:56.832445 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.832471 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:11:56.832505 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.835305 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:37131
I20251026 03:11:56.835355 7942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:37131 every 8 connection(s)
I20251026 03:11:56.838017 7943 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45221
I20251026 03:11:56.838064 7943 heartbeater.cc:461] Registering TS with master...
I20251026 03:11:56.838150 7943 heartbeater.cc:507] Master 127.30.128.126:45221 requested a full tablet report, sending...
I20251026 03:11:56.838323 7663 ts_manager.cc:194] Registered new tserver with Master: 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131)
I20251026 03:11:56.838614 7663 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33070
I20251026 03:11:56.838766 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00328731s
I20251026 03:11:56.839104 31233 test_util.cc:276] Using random seed: 624370880
I20251026 03:11:56.843055 7663 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33076:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:11:56.843169 7663 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:11:56.847551 7832 tablet_service.cc:1505] Processing CreateTablet for tablet 64628560ddff4d0b94db2b5502de12e5 (DEFAULT_TABLE table=test-workload [id=33b6cc56ebc2446da50a1ee0e341dec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:56.847669 7832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64628560ddff4d0b94db2b5502de12e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:56.847863 7907 tablet_service.cc:1505] Processing CreateTablet for tablet 64628560ddff4d0b94db2b5502de12e5 (DEFAULT_TABLE table=test-workload [id=33b6cc56ebc2446da50a1ee0e341dec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:56.847901 7756 tablet_service.cc:1505] Processing CreateTablet for tablet 64628560ddff4d0b94db2b5502de12e5 (DEFAULT_TABLE table=test-workload [id=33b6cc56ebc2446da50a1ee0e341dec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:11:56.848001 7907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64628560ddff4d0b94db2b5502de12e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:56.848011 7756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64628560ddff4d0b94db2b5502de12e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:11:56.849215 7959 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap starting.
I20251026 03:11:56.849326 7960 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap starting.
I20251026 03:11:56.849370 7958 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap starting.
I20251026 03:11:56.849622 7959 tablet_bootstrap.cc:654] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:56.849745 7958 tablet_bootstrap.cc:654] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:56.849745 7960 tablet_bootstrap.cc:654] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Neither blocks nor log segments found. Creating new log.
I20251026 03:11:56.850315 7958 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: No bootstrap required, opened a new log
I20251026 03:11:56.850354 7958 ts_tablet_manager.cc:1403] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251026 03:11:56.850405 7960 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: No bootstrap required, opened a new log
I20251026 03:11:56.850450 7960 ts_tablet_manager.cc:1403] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:11:56.850484 7958 raft_consensus.cc:359] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.850544 7958 raft_consensus.cc:385] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:56.850557 7958 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Initialized, Role: FOLLOWER
I20251026 03:11:56.850586 7960 raft_consensus.cc:359] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.850595 7958 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.850658 7960 raft_consensus.cc:385] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:56.850682 7960 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Initialized, Role: FOLLOWER
I20251026 03:11:56.850719 7959 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: No bootstrap required, opened a new log
I20251026 03:11:56.850749 7960 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.850766 7959 ts_tablet_manager.cc:1403] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251026 03:11:56.850764 7958 ts_tablet_manager.cc:1434] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:11:56.850930 7959 raft_consensus.cc:359] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.850919 7960 ts_tablet_manager.cc:1434] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:11:56.850914 7943 heartbeater.cc:499] Master 127.30.128.126:45221 was elected leader, sending a full tablet report...
I20251026 03:11:56.851006 7959 raft_consensus.cc:385] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:11:56.851030 7959 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Initialized, Role: FOLLOWER
I20251026 03:11:56.850911 7792 heartbeater.cc:499] Master 127.30.128.126:45221 was elected leader, sending a full tablet report...
I20251026 03:11:56.851089 7959 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.851265 7959 ts_tablet_manager.cc:1434] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:11:56.851326 7868 heartbeater.cc:499] Master 127.30.128.126:45221 was elected leader, sending a full tablet report...
I20251026 03:11:56.857760 7964 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:11:56.857844 7964 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.857987 7964 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:11:56.860061 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01" is_pre_election: true
I20251026 03:11:56.860141 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 0.
I20251026 03:11:56.860189 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a" is_pre_election: true
I20251026 03:11:56.860246 7842 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 0.
I20251026 03:11:56.860265 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters:
I20251026 03:11:56.860392 7964 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:11:56.860426 7964 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:11:56.860440 7964 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:56.860870 7964 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.860960 7964 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 1 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:11:56.861090 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:11:56.861114 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:11:56.861141 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:56.861152 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:11:56.861662 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 1.
I20251026 03:11:56.861728 7842 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 1.
I20251026 03:11:56.861829 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters:
I20251026 03:11:56.861919 7964 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:11:56.862018 7964 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 1 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:11:56.862056 7964 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:11:56.862495 7662 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:11:56.876307 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 1 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:56.876328 7842 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 1 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:11:56.876499 7964 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:11:56.876601 7968 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:12:06.926921 8013 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:06.926993 8013 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:12:06.927771 8013 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:06.928189 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 1. Current term is 2. Ops: []
I20251026 03:12:06.928339 8013 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 2 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:06.928646 8007 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20251026 03:12:06.928691 8007 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20251026 03:12:06.928906 8001 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:06.931017 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15906 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:06.931146 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15906 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:06.931291 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:06.931380 8013 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 03:12:06.952517 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 1. Current term is 2. Ops: [1.15907-1.15907]
I20251026 03:12:06.952672 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:06.958060 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 1. Current term is 2. Ops: [1.15907-1.15907]
I20251026 03:12:06.958307 8007 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:06.971714 8013 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:06.971760 8013 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:12:06.972218 8013 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:06.972312 8013 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 3 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:06.972422 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 15906 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:06.972429 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 3 candidate_status { last_received { term: 1 index: 15906 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:06.972645 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:06.972714 8013 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251026 03:12:06.985687 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 1. Current term is 3. Ops: [1.15907-1.15908]
I20251026 03:12:06.985893 8007 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:06.986029 7996 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:12:06.986101 7996 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:06.986159 7996 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15908, Committed index: 15908, Last appended: 1.15908, Last appended by leader: 15908, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:06.986233 7996 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:12:07.014586 8031 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:07.014631 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:12:07.015291 8031 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.015372 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.015429 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:12:07.015435 8031 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 2 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.016173 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.016283 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 3 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.016448 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 3 candidate_status { last_received { term: 1 index: 15908 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.016494 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 3 candidate_status { last_received { term: 1 index: 15908 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.016516 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 3: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:07.016536 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:12:07.017217 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 3.
I20251026 03:12:07.017390 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters:
I20251026 03:12:07.017465 7991 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 03:12:07.017503 7991 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 3 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:07.017545 7991 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15907, Committed index: 15907, Last appended: 1.15908, Last appended by leader: 15908, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.018100 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:07.018842 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 2 candidate_status { last_received { term: 1 index: 15908 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.018849 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 2 candidate_status { last_received { term: 1 index: 15908 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.018939 7841 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 2. Current term is 3.
I20251026 03:12:07.019130 7883 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 2 election: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 2. Current term is 3.
I20251026 03:12:07.019165 7883 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20251026 03:12:07.019227 8031 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 2. Current term is 3.
W20251026 03:12:07.024102 7996 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 3 is higher.
I20251026 03:12:07.040266 8013 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.040313 8013 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:12:07.040732 8013 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.040834 8013 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 4 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.040988 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 15906 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.040998 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 4 candidate_status { last_received { term: 1 index: 15906 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.041081 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:12:07.041633 7917 raft_consensus.cc:2410] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for term 4 because replica has last-logged OpId of term: 1 index: 15908, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 15906.
I20251026 03:12:07.041792 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.041886 8013 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 03:12:07.044143 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 4 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 4. Ops: []
I20251026 03:12:07.044333 8007 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15909, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:07.054448 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 4 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 4. Ops: []
I20251026 03:12:07.054625 8007 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15909, Last known committed idx: 15908, Time since last communication: 0.000s
I20251026 03:12:07.055419 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 4 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 4. Ops: [3.15909-3.15909]
I20251026 03:12:07.055469 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 4 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 4. Ops: [1.15907-3.15909]
I20251026 03:12:07.055601 8007 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 1.15908, Next index: 15909, Last known committed idx: 15908, Time since last communication: 0.000s
I20251026 03:12:07.055680 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:07.055929 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 4 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 4. Ops: [3.15909-3.15910]
I20251026 03:12:07.055934 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 4 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 4. Ops: [1.15907-3.15910]
I20251026 03:12:07.056092 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 1.15908, Next index: 15909, Last known committed idx: 15908, Time since last communication: 0.000s
I20251026 03:12:07.056150 8001 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:07.058574 8031 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.058619 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 4 FOLLOWER]: Advancing to term 5
I20251026 03:12:07.059088 8031 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.059185 8031 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 5 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.059316 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 5 candidate_status { last_received { term: 1 index: 15908 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.059394 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 4 FOLLOWER]: Advancing to term 5
I20251026 03:12:07.059372 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 5 candidate_status { last_received { term: 1 index: 15908 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.059932 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 5.
I20251026 03:12:07.060063 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040
I20251026 03:12:07.060142 8031 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 03:12:07.060250 8031 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 5 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:07.060292 8031 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15908, Committed index: 15908, Last appended: 1.15908, Last appended by leader: 15908, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.060688 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 3 to 5, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 5 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:07.085578 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 5 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 3. Current term is 5. Ops: [1.15907-3.15910]
I20251026 03:12:07.085762 8007 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 1.15906, Next index: 15907, Last known committed idx: 15906, Time since last communication: 0.000s
W20251026 03:12:07.087371 7991 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20251026 03:12:07.088022 7766 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 03:12:07.088069 7766 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:07.088116 7766 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15907, Committed index: 15907, Last appended: 3.15910, Last appended by leader: 15910, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.088193 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 3 FOLLOWER]: Advancing to term 5
I20251026 03:12:07.088702 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 5 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 3 index: 15910. Preceding OpId from leader: term: 5 index: 15909. (term mismatch)
I20251026 03:12:07.088739 7766 pending_rounds.cc:85] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Aborting all ops after (but not including) 15908
I20251026 03:12:07.088753 7766 pending_rounds.cc:107] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Aborting uncommitted NO_OP operation due to leader change: 3.15909
I20251026 03:12:07.088780 7766 raft_consensus.cc:2889] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 03:12:07.088816 7766 pending_rounds.cc:107] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Aborting uncommitted WRITE_OP operation due to leader change: 3.15910
I20251026 03:12:07.089071 8031 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15909, Last known committed idx: 15907, Time since last communication: 0.000s
I20251026 03:12:07.089586 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 5 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 1 index: 15906. Preceding OpId from leader: term: 5 index: 15910. (index mismatch)
I20251026 03:12:07.089794 8045 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15909, Last known committed idx: 15906, Time since last communication: 0.000s
I20251026 03:12:07.121361 8013 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 5 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:07.121414 8013 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 5 FOLLOWER]: Advancing to term 6
I20251026 03:12:07.122123 8013 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.122236 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 5. Current term is 6. Ops: []
I20251026 03:12:07.122269 8013 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 6 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.122403 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 5.15910, Next index: 15911, Last known committed idx: 15910, Time since last communication: 0.000s
I20251026 03:12:07.122412 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 15910 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.122386 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 6 candidate_status { last_received { term: 5 index: 15910 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.122633 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.122730 8013 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 03:12:07.144438 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 5. Current term is 6. Ops: []
I20251026 03:12:07.144611 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 5.15910, Next index: 15911, Last known committed idx: 15910, Time since last communication: 0.000s
I20251026 03:12:07.145244 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:07.145287 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 5 FOLLOWER]: Advancing to term 6
I20251026 03:12:07.145989 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.146092 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 6 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 5. Current term is 6. Ops: []
I20251026 03:12:07.146111 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 6 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.146242 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 6 candidate_status { last_received { term: 5 index: 15910 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.146292 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 6 candidate_status { last_received { term: 5 index: 15910 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.146359 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 6: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:07.146376 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 5.15910, Next index: 15911, Last known committed idx: 15910, Time since last communication: 0.000s
I20251026 03:12:07.146500 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.146588 8001 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 03:12:07.160584 8053 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 03:12:07.160629 8053 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:07.160684 8053 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15910, Committed index: 15910, Last appended: 5.15910, Last appended by leader: 15910, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.160745 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 5 FOLLOWER]: Advancing to term 6
I20251026 03:12:07.163012 8013 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.163050 8013 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 6 FOLLOWER]: Advancing to term 7
I20251026 03:12:07.163430 8013 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.163532 8013 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 7 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.163645 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 15910 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.163648 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 7 candidate_status { last_received { term: 5 index: 15910 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.163691 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 6 FOLLOWER]: Advancing to term 7
I20251026 03:12:07.163710 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 6 FOLLOWER]: Advancing to term 7
I20251026 03:12:07.164134 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 7.
I20251026 03:12:07.164250 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 7.
I20251026 03:12:07.164304 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters:
I20251026 03:12:07.164404 8013 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 7 FOLLOWER]: Leader election won for term 7
I20251026 03:12:07.164494 8013 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 7 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:07.164541 8013 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15910, Committed index: 15910, Last appended: 5.15910, Last appended by leader: 15910, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.164928 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 5 to 7, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 7 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:07.194212 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 7 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 15910. Preceding OpId from leader: term: 7 index: 15911. (index mismatch)
I20251026 03:12:07.194392 8013 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15911, Last known committed idx: 15910, Time since last communication: 0.000s
I20251026 03:12:07.195032 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 7 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 5 index: 15910. Preceding OpId from leader: term: 7 index: 15911. (index mismatch)
I20251026 03:12:07.195286 8057 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15911, Last known committed idx: 15910, Time since last communication: 0.000s
I20251026 03:12:07.226044 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:07.226094 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 7 FOLLOWER]: Advancing to term 8
I20251026 03:12:07.226553 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.226658 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 8 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.226774 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 8 candidate_status { last_received { term: 7 index: 15911 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.226812 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 8 candidate_status { last_received { term: 7 index: 15911 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.226972 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.227051 8001 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 03:12:07.227520 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 8 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 7. Current term is 8. Ops: []
I20251026 03:12:07.227685 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 7.15911, Next index: 15912, Last known committed idx: 15911, Time since last communication: 0.000s
I20251026 03:12:07.252053 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:07.252113 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 7 FOLLOWER]: Advancing to term 8
I20251026 03:12:07.252646 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.252735 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 8 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 7. Current term is 8. Ops: []
I20251026 03:12:07.252743 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 8 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.252871 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 8 candidate_status { last_received { term: 7 index: 15911 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.252915 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 8 candidate_status { last_received { term: 7 index: 15911 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.252925 8013 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 7.15911, Next index: 15912, Last known committed idx: 15911, Time since last communication: 0.000s
I20251026 03:12:07.252933 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 8: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:07.253142 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.253223 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20251026 03:12:07.255401 8057 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 7 LEADER]: Stepping down as leader of term 7
I20251026 03:12:07.255437 8057 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:07.255492 8057 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15911, Committed index: 15911, Last appended: 7.15911, Last appended by leader: 15911, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.255568 8057 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 7 FOLLOWER]: Advancing to term 8
I20251026 03:12:07.266331 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.266377 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 8 FOLLOWER]: Advancing to term 9
I20251026 03:12:07.266783 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.266912 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 9 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.267028 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 9 candidate_status { last_received { term: 7 index: 15911 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.267088 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 8 FOLLOWER]: Advancing to term 9
I20251026 03:12:07.267071 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 9 candidate_status { last_received { term: 7 index: 15911 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.267158 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 8 FOLLOWER]: Advancing to term 9
I20251026 03:12:07.267555 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 9.
I20251026 03:12:07.267632 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 9.
I20251026 03:12:07.267716 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters:
I20251026 03:12:07.267855 8001 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 9 FOLLOWER]: Leader election won for term 9
I20251026 03:12:07.267903 8001 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 9 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:07.267937 8001 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15911, Committed index: 15911, Last appended: 7.15911, Last appended by leader: 15911, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.268285 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 7 to 9, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 9 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
W20251026 03:12:07.271001 7991 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 9 is higher.
I20251026 03:12:07.296741 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 9 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 7 index: 15911. Preceding OpId from leader: term: 9 index: 15912. (index mismatch)
I20251026 03:12:07.296934 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15912, Last known committed idx: 15911, Time since last communication: 0.000s
I20251026 03:12:07.300724 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 9 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 7 index: 15911. Preceding OpId from leader: term: 9 index: 15912. (index mismatch)
I20251026 03:12:07.300925 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15912, Last known committed idx: 15911, Time since last communication: 0.000s
I20251026 03:12:07.327898 8053 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:07.327946 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 9 FOLLOWER]: Advancing to term 10
I20251026 03:12:07.328455 8053 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.328557 8053 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 10 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.328724 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 10 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.328753 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 10 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.328929 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.329066 8053 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251026 03:12:07.333179 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 10 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 9. Current term is 10. Ops: []
I20251026 03:12:07.333357 8001 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 9.15912, Next index: 15913, Last known committed idx: 15912, Time since last communication: 0.000s
I20251026 03:12:07.361549 8053 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.361595 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 10 FOLLOWER]: Advancing to term 11
I20251026 03:12:07.362110 8053 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.362201 8053 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 11 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.362344 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 11 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.362366 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 11 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.362509 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.362588 8053 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20251026 03:12:07.366427 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 9. Current term is 11. Ops: []
I20251026 03:12:07.366618 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 9.15912, Next index: 15913, Last known committed idx: 15912, Time since last communication: 0.000s
I20251026 03:12:07.385565 8057 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:07.385608 8057 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 9 FOLLOWER]: Advancing to term 10
I20251026 03:12:07.386086 8057 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.386178 8057 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 10 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.386317 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.386375 7917 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 10. Current term is 11.
I20251026 03:12:07.386322 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 10 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.386495 7806 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 10 election: Vote denied by peer 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 10. Current term is 11.
I20251026 03:12:07.386533 7806 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20251026 03:12:07.386675 8057 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 10 FOLLOWER]: Advancing to term 11
I20251026 03:12:07.387072 8057 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 10. Current term is 11.
I20251026 03:12:07.388792 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 11 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 9. Current term is 11. Ops: []
I20251026 03:12:07.389010 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 9.15912, Next index: 15913, Last known committed idx: 15912, Time since last communication: 0.000s
I20251026 03:12:07.394898 8053 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.394948 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 11 FOLLOWER]: Advancing to term 12
I20251026 03:12:07.395378 8053 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.395468 8053 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 12 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.395599 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 12 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.395642 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 11 FOLLOWER]: Advancing to term 12
I20251026 03:12:07.395612 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 12 candidate_status { last_received { term: 9 index: 15912 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.396085 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 12.
I20251026 03:12:07.396225 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040
I20251026 03:12:07.396310 8053 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 FOLLOWER]: Leader election won for term 12
I20251026 03:12:07.396365 8053 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:07.396410 8053 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15912, Committed index: 15912, Last appended: 9.15912, Last appended by leader: 15912, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.396747 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 9 to 12, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 12 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:07.397291 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 LEADER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 9. Current term is 12. Ops: []
I20251026 03:12:07.397464 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 9.15912, Next index: 15913, Last known committed idx: 15912, Time since last communication: 0.000s
I20251026 03:12:07.397532 7996 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer 360eb6050b7244d58989794aea84cc01 log is divergent from this leader: its last log entry 12.15913 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15912
I20251026 03:12:07.404938 8001 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 9 LEADER]: Stepping down as leader of term 9
I20251026 03:12:07.404984 8001 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:07.405040 8001 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15912, Committed index: 15912, Last appended: 9.15912, Last appended by leader: 15912, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.405107 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 9 FOLLOWER]: Advancing to term 10
I20251026 03:12:07.419013 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 10 FOLLOWER]: Advancing to term 12
I20251026 03:12:07.419477 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 12 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 9 index: 15912. Preceding OpId from leader: term: 12 index: 15913. (index mismatch)
I20251026 03:12:07.419685 8053 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15913, Last known committed idx: 15912, Time since last communication: 0.000s
I20251026 03:12:07.420861 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 12 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 9 index: 15912. Preceding OpId from leader: term: 12 index: 15913. (index mismatch)
I20251026 03:12:07.421031 8053 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15913, Last known committed idx: 15912, Time since last communication: 0.000s
I20251026 03:12:07.450933 7996 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:07.450986 7996 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 12 FOLLOWER]: Advancing to term 13
I20251026 03:12:07.451447 7996 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.451566 7996 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 13 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.451516 8057 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 12 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:07.451622 8057 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 12 FOLLOWER]: Advancing to term 13
I20251026 03:12:07.451669 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 13 candidate_status { last_received { term: 12 index: 15913 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.451678 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 13 candidate_status { last_received { term: 12 index: 15913 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.452205 8057 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.452328 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 13: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:07.452401 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 15913 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.452420 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 13 candidate_status { last_received { term: 12 index: 15913 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.452477 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 13: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:07.452482 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.452562 8057 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 13 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.452585 7996 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20251026 03:12:07.452648 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.452724 8057 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20251026 03:12:07.454807 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 13 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 12. Current term is 13. Ops: []
I20251026 03:12:07.454977 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 12.15913, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.456140 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 12. Current term is 13. Ops: []
I20251026 03:12:07.456310 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 12.15913, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.479401 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 12. Current term is 13. Ops: []
I20251026 03:12:07.479570 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 12.15913, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.484330 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 13 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 12. Current term is 13. Ops: []
I20251026 03:12:07.484534 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 12.15913, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.493224 8057 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.493271 8057 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 13 FOLLOWER]: Advancing to term 14
I20251026 03:12:07.493692 8057 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.493865 8057 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 14 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.494001 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 15913 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.494019 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 14 candidate_status { last_received { term: 12 index: 15913 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.494079 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 13 FOLLOWER]: Advancing to term 14
I20251026 03:12:07.494535 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 14.
I20251026 03:12:07.494674 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.494750 8057 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 FOLLOWER]: Leader election won for term 14
I20251026 03:12:07.494796 8057 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:07.494833 8057 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15913, Committed index: 15913, Last appended: 12.15913, Last appended by leader: 15913, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.495158 7656 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 12 to 14, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 14 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:07.502569 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 LEADER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 12. Current term is 14. Ops: []
I20251026 03:12:07.502740 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 12.15913, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.502821 8031 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer 5340c78feac8406882e860a2bed2ec2a log is divergent from this leader: its last log entry 14.15914 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15913
I20251026 03:12:07.508566 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 14 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 12. Current term is 14. Ops: []
I20251026 03:12:07.508780 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 12.15913, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.525800 8045 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 LEADER]: Stepping down as leader of term 12
I20251026 03:12:07.525867 8045 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:07.526021 8045 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15913, Committed index: 15913, Last appended: 12.15913, Last appended by leader: 15913, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.526114 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 12 FOLLOWER]: Advancing to term 13
I20251026 03:12:07.527382 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 14 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 12 index: 15913. Preceding OpId from leader: term: 14 index: 15914. (index mismatch)
I20251026 03:12:07.527570 8057 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.529431 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 13 FOLLOWER]: Advancing to term 14
I20251026 03:12:07.529884 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 14 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 12 index: 15913. Preceding OpId from leader: term: 14 index: 15914. (index mismatch)
I20251026 03:12:07.530071 8057 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15914, Last known committed idx: 15913, Time since last communication: 0.000s
I20251026 03:12:07.558697 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:07.558744 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 14 FOLLOWER]: Advancing to term 15
I20251026 03:12:07.559178 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.559270 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 15 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.559402 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 15 candidate_status { last_received { term: 14 index: 15914 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.559407 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 15 candidate_status { last_received { term: 14 index: 15914 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.559600 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.559669 8001 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20251026 03:12:07.562909 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 14. Current term is 15. Ops: []
I20251026 03:12:07.563094 8057 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 14.15914, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.583695 8031 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:07.583793 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 14 FOLLOWER]: Advancing to term 15
I20251026 03:12:07.584499 8031 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.584614 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 15 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 14. Current term is 15. Ops: []
I20251026 03:12:07.584620 8031 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 15 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.584931 8013 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 14.15914, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.584956 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 15 candidate_status { last_received { term: 14 index: 15914 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.585094 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 15 candidate_status { last_received { term: 14 index: 15914 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.585150 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 15: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:07.585283 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.585358 8031 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20251026 03:12:07.586272 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 14. Current term is 15. Ops: []
I20251026 03:12:07.586443 8057 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 14.15914, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.598376 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.598425 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 15 FOLLOWER]: Advancing to term 16
I20251026 03:12:07.598845 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.598941 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 16 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.599107 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 16 candidate_status { last_received { term: 14 index: 15914 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.599118 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 16 candidate_status { last_received { term: 14 index: 15914 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.599167 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 15 FOLLOWER]: Advancing to term 16
I20251026 03:12:07.599676 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 16.
I20251026 03:12:07.599833 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters: 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.599926 8001 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 FOLLOWER]: Leader election won for term 16
I20251026 03:12:07.599967 8001 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:07.600005 8001 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15914, Committed index: 15914, Last appended: 14.15914, Last appended by leader: 15914, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.600476 7656 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 14 to 16, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 16 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:07.616741 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 LEADER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 14. Current term is 16. Ops: []
I20251026 03:12:07.616912 8013 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 14.15914, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.616964 8013 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer 2a26a9c2ccd44b42a5d55fe3c9607040 log is divergent from this leader: its last log entry 16.15915 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15914
I20251026 03:12:07.619942 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 16 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 14. Current term is 16. Ops: []
I20251026 03:12:07.620116 8013 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 14.15914, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.623126 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 16 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 14 index: 15914. Preceding OpId from leader: term: 16 index: 15915. (index mismatch)
I20251026 03:12:07.623291 7996 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.632486 7841 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 LEADER]: Stepping down as leader of term 14
I20251026 03:12:07.632524 7841 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:07.632555 7841 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15914, Committed index: 15914, Last appended: 14.15914, Last appended by leader: 15914, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.632625 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 14 FOLLOWER]: Advancing to term 16
I20251026 03:12:07.633039 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 16 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 14 index: 15914. Preceding OpId from leader: term: 16 index: 15915. (index mismatch)
I20251026 03:12:07.633224 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15915, Last known committed idx: 15914, Time since last communication: 0.000s
I20251026 03:12:07.705317 8031 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:07.705400 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 16 FOLLOWER]: Advancing to term 17
I20251026 03:12:07.706022 8031 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.706156 8031 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 17 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.706332 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 17 candidate_status { last_received { term: 16 index: 15915 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.706332 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 17 candidate_status { last_received { term: 16 index: 15915 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.706568 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:07.706697 8031 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20251026 03:12:07.708122 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 17 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 16. Current term is 17. Ops: []
I20251026 03:12:07.708315 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 16.15915, Next index: 15916, Last known committed idx: 15915, Time since last communication: 0.000s
I20251026 03:12:07.717857 8057 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 16 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:07.717916 8057 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 16 FOLLOWER]: Advancing to term 17
I20251026 03:12:07.718653 8057 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.718789 8057 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 17 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.718962 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 15915 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.718971 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 17 candidate_status { last_received { term: 16 index: 15915 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.719118 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 17: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:07.719236 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.719349 8057 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20251026 03:12:07.721434 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 17 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 16. Current term is 17. Ops: []
I20251026 03:12:07.721643 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 16.15915, Next index: 15916, Last known committed idx: 15915, Time since last communication: 0.000s
I20251026 03:12:07.725796 8001 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 LEADER]: Stepping down as leader of term 16
I20251026 03:12:07.725843 8001 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:07.725885 8001 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15915, Committed index: 15915, Last appended: 16.15915, Last appended by leader: 15915, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.725943 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 16 FOLLOWER]: Advancing to term 17
I20251026 03:12:07.741066 8031 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.741115 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 17 FOLLOWER]: Advancing to term 18
I20251026 03:12:07.741614 8031 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.741714 8031 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 18 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.741875 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 18 candidate_status { last_received { term: 16 index: 15915 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.741907 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 18 candidate_status { last_received { term: 16 index: 15915 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.741954 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 17 FOLLOWER]: Advancing to term 18
I20251026 03:12:07.741959 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 17 FOLLOWER]: Advancing to term 18
I20251026 03:12:07.742566 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 18.
I20251026 03:12:07.742566 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 18.
I20251026 03:12:07.742759 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters:
I20251026 03:12:07.742897 8031 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 18 FOLLOWER]: Leader election won for term 18
I20251026 03:12:07.742949 8031 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 18 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:07.742988 8031 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15915, Committed index: 15915, Last appended: 16.15915, Last appended by leader: 15915, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.743403 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 16 to 18, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 18 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:07.771845 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 18 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 16 index: 15915. Preceding OpId from leader: term: 18 index: 15916. (index mismatch)
I20251026 03:12:07.772065 8031 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15916, Last known committed idx: 15915, Time since last communication: 0.000s
I20251026 03:12:07.772408 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 18 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 16 index: 15915. Preceding OpId from leader: term: 18 index: 15916. (index mismatch)
I20251026 03:12:07.772559 8031 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15916, Last known committed idx: 15915, Time since last communication: 0.000s
I20251026 03:12:07.803966 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 18 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:07.804031 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 18 FOLLOWER]: Advancing to term 19
I20251026 03:12:07.804693 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.804828 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 19 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.805022 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.805033 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.805233 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.805320 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20251026 03:12:07.806488 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 19 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 18. Current term is 19. Ops: []
I20251026 03:12:07.806667 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 18.15916, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:07.842043 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 19 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 18. Current term is 19. Ops: []
I20251026 03:12:07.842283 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 18.15916, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:07.846307 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.846359 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 19 FOLLOWER]: Advancing to term 20
I20251026 03:12:07.846932 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.847059 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 20 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.847188 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.847198 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 20 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.847396 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:07.847483 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20251026 03:12:07.874285 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 18. Current term is 20. Ops: []
I20251026 03:12:07.874521 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 18.15916, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:07.882783 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:07.882864 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 18 FOLLOWER]: Advancing to term 19
I20251026 03:12:07.883394 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.883499 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 19 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 18. Current term is 19. Ops: []
I20251026 03:12:07.883522 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 19 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:07.883634 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 19 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:07.883644 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 19 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.883682 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 18.15916, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:07.883745 7841 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 19. Current term is 20.
I20251026 03:12:07.883972 7732 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 19 election: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 19. Current term is 20.
I20251026 03:12:07.884009 7732 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20251026 03:12:07.884096 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 19 FOLLOWER]: Advancing to term 20
I20251026 03:12:07.884575 8001 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 19. Current term is 20.
I20251026 03:12:07.899487 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 18. Current term is 20. Ops: []
I20251026 03:12:07.899701 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 18.15916, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:07.903455 8031 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 18 LEADER]: Stepping down as leader of term 18
I20251026 03:12:07.903498 8031 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:07.903558 8031 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15916, Committed index: 15916, Last appended: 18.15916, Last appended by leader: 15916, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.903640 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 18 FOLLOWER]: Advancing to term 19
I20251026 03:12:07.913031 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:07.913085 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 20 FOLLOWER]: Advancing to term 21
I20251026 03:12:07.913559 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.913667 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 21 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:07.913846 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:07.913879 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 21 candidate_status { last_received { term: 18 index: 15916 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:07.913915 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 19 FOLLOWER]: Advancing to term 21
I20251026 03:12:07.913933 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 20 FOLLOWER]: Advancing to term 21
I20251026 03:12:07.914397 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 21.
I20251026 03:12:07.914450 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 21.
I20251026 03:12:07.914577 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters:
I20251026 03:12:07.914670 8058 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 21 FOLLOWER]: Leader election won for term 21
I20251026 03:12:07.914731 8058 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 21 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:07.914772 8058 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15916, Committed index: 15916, Last appended: 18.15916, Last appended by leader: 15916, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:07.915256 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 18 to 21, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 21 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:07.944411 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 21 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15916. Preceding OpId from leader: term: 21 index: 15917. (index mismatch)
I20251026 03:12:07.944638 8058 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:07.947857 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 21 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15916. Preceding OpId from leader: term: 21 index: 15917. (index mismatch)
I20251026 03:12:07.948089 8013 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15917, Last known committed idx: 15916, Time since last communication: 0.000s
I20251026 03:12:08.004305 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:08.004384 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 21 FOLLOWER]: Advancing to term 22
I20251026 03:12:08.005054 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.005189 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 22 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.005353 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 22 candidate_status { last_received { term: 21 index: 15917 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.005379 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 22 candidate_status { last_received { term: 21 index: 15917 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.005605 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.005749 8001 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20251026 03:12:08.007292 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 22 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 21. Current term is 22. Ops: []
I20251026 03:12:08.007501 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 21.15917, Next index: 15918, Last known committed idx: 15917, Time since last communication: 0.000s
I20251026 03:12:08.022691 8058 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 21 LEADER]: Stepping down as leader of term 21
I20251026 03:12:08.022753 8058 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:08.022823 8058 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15917, Committed index: 15917, Last appended: 21.15917, Last appended by leader: 15917, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.022913 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 21 FOLLOWER]: Advancing to term 22
I20251026 03:12:08.039786 8031 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:08.039857 8031 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 21 FOLLOWER]: Advancing to term 22
I20251026 03:12:08.040413 8031 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.040560 8031 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 22 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.040791 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 22 candidate_status { last_received { term: 21 index: 15917 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.040791 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 22 candidate_status { last_received { term: 21 index: 15917 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.040915 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 22: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:08.041435 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 22.
I20251026 03:12:08.041611 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040
I20251026 03:12:08.041697 8031 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 22 FOLLOWER]: Leader election won for term 22
I20251026 03:12:08.041759 8031 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 22 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:08.041841 8031 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15917, Committed index: 15917, Last appended: 21.15917, Last appended by leader: 15917, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.042270 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 21 to 22, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 22 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:08.048159 8001 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.048213 8001 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 22 FOLLOWER]: Advancing to term 23
I20251026 03:12:08.048655 8001 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.048763 8001 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 23 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.048919 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 23 candidate_status { last_received { term: 21 index: 15917 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.048936 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 23 candidate_status { last_received { term: 21 index: 15917 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.048988 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 22 FOLLOWER]: Advancing to term 23
I20251026 03:12:08.049486 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 23.
I20251026 03:12:08.049660 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.049748 8001 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 FOLLOWER]: Leader election won for term 23
I20251026 03:12:08.049819 8001 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:08.049870 8001 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15917, Committed index: 15917, Last appended: 21.15917, Last appended by leader: 15917, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.052026 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 22 to 23, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 23 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:08.069604 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 LEADER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 22. Current term is 23. Ops: []
I20251026 03:12:08.069850 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15918, Last known committed idx: 15917, Time since last communication: 0.000s
I20251026 03:12:08.069927 8031 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer 2a26a9c2ccd44b42a5d55fe3c9607040 log is divergent from this leader: its last log entry 23.15918 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15917
I20251026 03:12:08.075397 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 23 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 22. Current term is 23. Ops: []
I20251026 03:12:08.075431 7917 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 22 LEADER]: Stepping down as leader of term 22
I20251026 03:12:08.075464 7917 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:08.075513 7917 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15917, Committed index: 15917, Last appended: 22.15918, Last appended by leader: 15918, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.075596 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 22 FOLLOWER]: Advancing to term 23
W20251026 03:12:08.075596 8031 consensus_queue.cc:1175] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5340c78feac8406882e860a2bed2ec2a" responder_term: 23 status { last_received { term: 21 index: 15917 } last_committed_idx: 15917 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 22. Current term is 23. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20251026 03:12:08.076182 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 23 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 21 index: 15917. Preceding OpId from leader: term: 23 index: 15918. (index mismatch)
I20251026 03:12:08.076315 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 23 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 22 index: 15918. Preceding OpId from leader: term: 23 index: 15918. (term mismatch)
I20251026 03:12:08.076361 7917 pending_rounds.cc:85] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting all ops after (but not including) 15917
I20251026 03:12:08.076388 7917 pending_rounds.cc:107] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting uncommitted NO_OP operation due to leader change: 22.15918
I20251026 03:12:08.076363 8001 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15918, Last known committed idx: 15917, Time since last communication: 0.000s
I20251026 03:12:08.076452 7917 raft_consensus.cc:2889] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 03:12:08.076650 8001 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15918, Last known committed idx: 15917, Time since last communication: 0.000s
W20251026 03:12:08.149762 7976 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
I20251026 03:12:08.149937 7976 mvcc.cc:204] Tried to move back new op lower bound from 7214892347757748224 to 7214892347639558144. Current Snapshot: MvccSnapshot[applied={T|T < 7214892347757748224}]
I20251026 03:12:08.181293 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:08.181370 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 23 FOLLOWER]: Advancing to term 24
I20251026 03:12:08.182034 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.182181 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 24 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.182343 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 24 candidate_status { last_received { term: 23 index: 15918 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.182384 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 24 candidate_status { last_received { term: 23 index: 15918 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.182575 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.182651 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20251026 03:12:08.183470 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 23. Current term is 24. Ops: []
I20251026 03:12:08.183660 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 23.15918, Next index: 15919, Last known committed idx: 15918, Time since last communication: 0.000s
I20251026 03:12:08.190030 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 23 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:08.190102 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 23 FOLLOWER]: Advancing to term 24
I20251026 03:12:08.190590 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.190694 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 24 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 23. Current term is 24. Ops: []
I20251026 03:12:08.190709 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 24 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.190827 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15918 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.190840 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 24 candidate_status { last_received { term: 23 index: 15918 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.190866 8001 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 23.15918, Next index: 15919, Last known committed idx: 15918, Time since last communication: 0.000s
I20251026 03:12:08.190933 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 24: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:08.191048 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.191150 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20251026 03:12:08.208328 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 23. Current term is 24. Ops: []
I20251026 03:12:08.208555 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 23.15918, Next index: 15919, Last known committed idx: 15918, Time since last communication: 0.000s
I20251026 03:12:08.219972 7991 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 LEADER]: Stepping down as leader of term 23
I20251026 03:12:08.220022 7991 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:08.220088 7991 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15918, Committed index: 15918, Last appended: 23.15918, Last appended by leader: 15918, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.220170 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 23 FOLLOWER]: Advancing to term 24
I20251026 03:12:08.220971 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.221016 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 24 FOLLOWER]: Advancing to term 25
I20251026 03:12:08.221531 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.221635 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 25 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.221804 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 25 candidate_status { last_received { term: 23 index: 15918 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.221813 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 25 candidate_status { last_received { term: 23 index: 15918 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.221884 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 24 FOLLOWER]: Advancing to term 25
I20251026 03:12:08.221889 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 24 FOLLOWER]: Advancing to term 25
I20251026 03:12:08.222457 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 25.
I20251026 03:12:08.222457 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 25.
I20251026 03:12:08.222646 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters:
I20251026 03:12:08.222734 8045 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 25 FOLLOWER]: Leader election won for term 25
I20251026 03:12:08.222784 8045 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 25 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:08.222834 8045 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15918, Committed index: 15918, Last appended: 23.15918, Last appended by leader: 15918, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.223194 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 23 to 25, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 25 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:08.251549 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 25 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 23 index: 15918. Preceding OpId from leader: term: 25 index: 15919. (index mismatch)
I20251026 03:12:08.251788 8045 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15919, Last known committed idx: 15918, Time since last communication: 0.000s
I20251026 03:12:08.254537 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 25 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 23 index: 15918. Preceding OpId from leader: term: 25 index: 15919. (index mismatch)
I20251026 03:12:08.254711 8053 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15919, Last known committed idx: 15918, Time since last communication: 0.000s
I20251026 03:12:08.283710 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:08.283835 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 25 FOLLOWER]: Advancing to term 26
I20251026 03:12:08.284497 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.284644 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 26 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.284852 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 26 candidate_status { last_received { term: 25 index: 15919 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.284906 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 25. Current term is 26. Ops: []
I20251026 03:12:08.284868 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 26 candidate_status { last_received { term: 25 index: 15919 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.284978 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 25 FOLLOWER]: Advancing to term 26
I20251026 03:12:08.285187 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 25.15919, Next index: 15920, Last known committed idx: 15919, Time since last communication: 0.000s
I20251026 03:12:08.285606 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 26.
I20251026 03:12:08.285830 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.285925 7991 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 FOLLOWER]: Leader election won for term 26
I20251026 03:12:08.285969 7991 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:08.286033 7991 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15919, Committed index: 15919, Last appended: 25.15919, Last appended by leader: 15919, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.286448 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 25 to 26, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 26 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:08.291846 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 26 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 25. Current term is 26. Ops: []
I20251026 03:12:08.292014 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 25.15919, Next index: 15920, Last known committed idx: 15919, Time since last communication: 0.000s
I20251026 03:12:08.308239 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 LEADER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 25. Current term is 26. Ops: []
I20251026 03:12:08.308480 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 25.15919, Next index: 15920, Last known committed idx: 15919, Time since last communication: 0.000s
I20251026 03:12:08.308553 8045 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer 2a26a9c2ccd44b42a5d55fe3c9607040 log is divergent from this leader: its last log entry 26.15920 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15919
I20251026 03:12:08.309691 7917 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 25 LEADER]: Stepping down as leader of term 25
I20251026 03:12:08.309723 7917 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:08.309782 7917 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15919, Committed index: 15919, Last appended: 25.15919, Last appended by leader: 15919, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.309861 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 25 FOLLOWER]: Advancing to term 26
I20251026 03:12:08.310361 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 26 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 25 index: 15919. Preceding OpId from leader: term: 26 index: 15920. (index mismatch)
I20251026 03:12:08.310554 8001 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15920, Last known committed idx: 15919, Time since last communication: 0.000s
I20251026 03:12:08.311419 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 26 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 25 index: 15919. Preceding OpId from leader: term: 26 index: 15920. (index mismatch)
I20251026 03:12:08.311574 8001 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15920, Last known committed idx: 15919, Time since last communication: 0.000s
I20251026 03:12:08.342070 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:08.342136 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 26 FOLLOWER]: Advancing to term 27
I20251026 03:12:08.342723 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.342857 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 27 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.343047 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 27 candidate_status { last_received { term: 26 index: 15920 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.343087 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 27 candidate_status { last_received { term: 26 index: 15920 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.343279 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.343381 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20251026 03:12:08.346442 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 27 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 26. Current term is 27. Ops: []
I20251026 03:12:08.346668 8001 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 26.15920, Next index: 15921, Last known committed idx: 15920, Time since last communication: 0.000s
I20251026 03:12:08.374415 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 27 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 26. Current term is 27. Ops: []
I20251026 03:12:08.374617 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 26.15920, Next index: 15921, Last known committed idx: 15920, Time since last communication: 0.000s
I20251026 03:12:08.378381 7991 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 LEADER]: Stepping down as leader of term 26
I20251026 03:12:08.378427 7991 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:08.378494 7991 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15920, Committed index: 15920, Last appended: 26.15920, Last appended by leader: 15920, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.378590 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 26 FOLLOWER]: Advancing to term 27
I20251026 03:12:08.384721 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.384778 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 27 FOLLOWER]: Advancing to term 28
I20251026 03:12:08.385268 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.385365 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 28 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.385550 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 28 candidate_status { last_received { term: 26 index: 15920 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.385543 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 28 candidate_status { last_received { term: 26 index: 15920 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.385635 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 27 FOLLOWER]: Advancing to term 28
I20251026 03:12:08.386119 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 28.
I20251026 03:12:08.386288 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters: 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.386395 8045 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 28 FOLLOWER]: Leader election won for term 28
I20251026 03:12:08.386456 8045 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 28 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:08.386505 8045 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15920, Committed index: 15920, Last appended: 26.15920, Last appended by leader: 15920, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.386914 7656 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 26 to 28, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 28 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:08.397671 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 26 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:08.397727 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 26 FOLLOWER]: Advancing to term 27
I20251026 03:12:08.398267 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.398381 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 27 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.398545 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 15920 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.398607 7766 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 27. Current term is 28.
I20251026 03:12:08.398540 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 27 candidate_status { last_received { term: 26 index: 15920 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.398713 7808 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 27 election: Vote denied by peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 27. Current term is 28.
I20251026 03:12:08.398753 7808 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20251026 03:12:08.398842 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 27 FOLLOWER]: Advancing to term 28
I20251026 03:12:08.399272 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 27. Current term is 28.
W20251026 03:12:08.402340 8031 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 28 is higher.
I20251026 03:12:08.416195 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 28 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 26 index: 15920. Preceding OpId from leader: term: 28 index: 15921. (index mismatch)
I20251026 03:12:08.416409 8031 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15921, Last known committed idx: 15920, Time since last communication: 0.000s
I20251026 03:12:08.417176 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 26 index: 15920. Preceding OpId from leader: term: 28 index: 15921. (index mismatch)
I20251026 03:12:08.417450 8053 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15921, Last known committed idx: 15920, Time since last communication: 0.000s
I20251026 03:12:08.448037 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:08.448133 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 28 FOLLOWER]: Advancing to term 29
I20251026 03:12:08.448760 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.448920 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 29 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.449132 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 29 candidate_status { last_received { term: 28 index: 15921 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.449138 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 29 candidate_status { last_received { term: 28 index: 15921 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.449347 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.449446 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20251026 03:12:08.453886 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 28. Current term is 29. Ops: []
I20251026 03:12:08.454074 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 28.15921, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.471235 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 28 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:08.471299 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 28 FOLLOWER]: Advancing to term 29
I20251026 03:12:08.471889 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.472046 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 29 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.472226 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 15921 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.472232 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 29 candidate_status { last_received { term: 28 index: 15921 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.472311 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 29: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:08.472438 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.472538 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20251026 03:12:08.477173 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 29 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 28. Current term is 29. Ops: []
I20251026 03:12:08.477363 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 28.15921, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.481279 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 28. Current term is 29. Ops: []
I20251026 03:12:08.481442 8031 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 28.15921, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.493737 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.493818 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 29 FOLLOWER]: Advancing to term 30
I20251026 03:12:08.494372 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.494495 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 30 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.494628 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 30 candidate_status { last_received { term: 28 index: 15921 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.494666 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 30 candidate_status { last_received { term: 28 index: 15921 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.494721 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 29 FOLLOWER]: Advancing to term 30
I20251026 03:12:08.495270 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 30.
I20251026 03:12:08.495457 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.495543 7991 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 FOLLOWER]: Leader election won for term 30
I20251026 03:12:08.495601 7991 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:08.495652 7991 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15921, Committed index: 15921, Last appended: 28.15921, Last appended by leader: 15921, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.496064 7656 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 28 to 30, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 30 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:08.509675 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 30 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 28. Current term is 30. Ops: []
I20251026 03:12:08.509928 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 28.15921, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.512773 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 LEADER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 28. Current term is 30. Ops: []
I20251026 03:12:08.512923 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 28.15921, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.512988 8053 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer 2a26a9c2ccd44b42a5d55fe3c9607040 log is divergent from this leader: its last log entry 30.15922 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15921
I20251026 03:12:08.517731 8045 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 28 LEADER]: Stepping down as leader of term 28
I20251026 03:12:08.517800 8045 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:08.517876 8045 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15921, Committed index: 15921, Last appended: 28.15921, Last appended by leader: 15921, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.517966 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 28 FOLLOWER]: Advancing to term 29
I20251026 03:12:08.525269 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 30 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 28 index: 15921. Preceding OpId from leader: term: 30 index: 15922. (index mismatch)
I20251026 03:12:08.525309 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 29 FOLLOWER]: Advancing to term 30
I20251026 03:12:08.525456 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.525906 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 30 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 28 index: 15921. Preceding OpId from leader: term: 30 index: 15922. (index mismatch)
I20251026 03:12:08.526070 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15922, Last known committed idx: 15921, Time since last communication: 0.000s
I20251026 03:12:08.557147 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 30 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:08.557204 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 30 FOLLOWER]: Advancing to term 31
I20251026 03:12:08.557675 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.557806 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 31 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.557971 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.557966 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 31 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.558192 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.558333 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20251026 03:12:08.560426 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 31 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 30. Current term is 31. Ops: []
I20251026 03:12:08.560606 7996 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 30.15922, Next index: 15923, Last known committed idx: 15922, Time since last communication: 0.000s
I20251026 03:12:08.591841 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 31 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 30. Current term is 31. Ops: []
I20251026 03:12:08.592051 8001 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 30.15922, Next index: 15923, Last known committed idx: 15922, Time since last communication: 0.000s
I20251026 03:12:08.592615 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.592648 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 31 FOLLOWER]: Advancing to term 32
I20251026 03:12:08.593137 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.593250 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 32 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.593369 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 32 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.593397 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 32 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.593545 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.593626 8058 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20251026 03:12:08.607642 7991 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 LEADER]: Stepping down as leader of term 30
I20251026 03:12:08.607685 7991 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:08.607723 7991 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15922, Committed index: 15922, Last appended: 30.15922, Last appended by leader: 15922, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.607807 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 30 FOLLOWER]: Advancing to term 31
I20251026 03:12:08.628949 8053 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:08.629032 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 30 FOLLOWER]: Advancing to term 31
I20251026 03:12:08.629603 8053 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.629745 8053 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 31 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.629966 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 31 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.629949 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 31 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.630087 7841 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 31. Current term is 32.
I20251026 03:12:08.630259 7883 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 31 election: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 31. Current term is 32.
I20251026 03:12:08.630292 7883 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20251026 03:12:08.630342 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 31 FOLLOWER]: Advancing to term 32
I20251026 03:12:08.630690 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 31.
I20251026 03:12:08.630822 8053 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 31. Current term is 32.
I20251026 03:12:08.653540 8058 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.653617 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 32 FOLLOWER]: Advancing to term 33
I20251026 03:12:08.654215 8058 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.654383 8058 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 33 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.654567 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 33 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.654567 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 33 candidate_status { last_received { term: 30 index: 15922 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.654667 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 31 FOLLOWER]: Advancing to term 33
I20251026 03:12:08.654699 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 32 FOLLOWER]: Advancing to term 33
I20251026 03:12:08.655247 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 33.
I20251026 03:12:08.655290 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 33.
I20251026 03:12:08.655413 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters:
I20251026 03:12:08.655499 8058 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 33 FOLLOWER]: Leader election won for term 33
I20251026 03:12:08.655608 8058 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 33 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:08.655689 8058 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15922, Committed index: 15922, Last appended: 30.15922, Last appended by leader: 15922, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.656091 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 30 to 33, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 33 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:08.686295 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 33 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 30 index: 15922. Preceding OpId from leader: term: 33 index: 15923. (index mismatch)
I20251026 03:12:08.686302 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 33 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 30 index: 15922. Preceding OpId from leader: term: 33 index: 15923. (index mismatch)
I20251026 03:12:08.686514 8065 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15923, Last known committed idx: 15922, Time since last communication: 0.000s
I20251026 03:12:08.686607 8058 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15923, Last known committed idx: 15922, Time since last communication: 0.000s
I20251026 03:12:08.744382 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:08.744453 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 33 FOLLOWER]: Advancing to term 34
I20251026 03:12:08.745036 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.745199 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 34 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.745361 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 34 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.745371 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 34 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.745574 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.745671 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20251026 03:12:08.747242 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:08.747277 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 33 FOLLOWER]: Advancing to term 34
I20251026 03:12:08.747860 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.747995 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 34 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.748118 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 34 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.748183 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 34 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.748253 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 34: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:08.748371 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.748451 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20251026 03:12:08.748557 7766 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 34 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 33. Current term is 34. Ops: []
I20251026 03:12:08.748615 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 34 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 33. Current term is 34. Ops: []
I20251026 03:12:08.748754 8066 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 33.15923, Next index: 15924, Last known committed idx: 15923, Time since last communication: 0.000s
I20251026 03:12:08.748868 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 33.15923, Next index: 15924, Last known committed idx: 15923, Time since last communication: 0.000s
I20251026 03:12:08.760059 8066 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 33 LEADER]: Stepping down as leader of term 33
I20251026 03:12:08.760120 8066 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:08.760195 8066 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15923, Committed index: 15923, Last appended: 33.15923, Last appended by leader: 15923, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.760293 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 33 FOLLOWER]: Advancing to term 34
I20251026 03:12:08.783228 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.783288 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 34 FOLLOWER]: Advancing to term 35
I20251026 03:12:08.783521 8066 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.783567 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 34 FOLLOWER]: Advancing to term 35
I20251026 03:12:08.783877 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.783947 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 34 FOLLOWER]: Advancing to term 35
I20251026 03:12:08.783892 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.784082 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 35 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.784248 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 35 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.784241 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 35 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.784245 8066 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.784363 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 35: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:08.784428 8066 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 35 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.784559 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 35 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.784593 7916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 35 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.784659 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 35: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:08.784792 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:08.784904 8066 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20251026 03:12:08.785055 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.785152 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 35 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.785140 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 35: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:08.785238 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 35 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.785271 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 35: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:08.785303 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 35 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.785336 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.785359 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 35: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:08.785414 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20251026 03:12:08.785475 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.785557 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20251026 03:12:08.815341 8066 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.815414 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 35 FOLLOWER]: Advancing to term 36
I20251026 03:12:08.815938 8066 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.816066 8066 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 36 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.816262 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 36 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.816262 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 36 candidate_status { last_received { term: 33 index: 15923 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.816356 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 35 FOLLOWER]: Advancing to term 36
I20251026 03:12:08.816363 7766 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 35 FOLLOWER]: Advancing to term 36
I20251026 03:12:08.816975 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 36.
I20251026 03:12:08.816975 7766 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 36.
I20251026 03:12:08.817175 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters:
I20251026 03:12:08.817283 8066 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 36 FOLLOWER]: Leader election won for term 36
I20251026 03:12:08.817353 8066 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 36 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:08.817394 8066 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15923, Committed index: 15923, Last appended: 33.15923, Last appended by leader: 15923, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.817831 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 33 to 36. New cstate: current_term: 36 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:08.847304 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 36 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 33 index: 15923. Preceding OpId from leader: term: 36 index: 15924. (index mismatch)
I20251026 03:12:08.847527 8066 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15924, Last known committed idx: 15923, Time since last communication: 0.000s
I20251026 03:12:08.847812 7766 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 36 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 33 index: 15923. Preceding OpId from leader: term: 36 index: 15924. (index mismatch)
I20251026 03:12:08.848048 8058 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15924, Last known committed idx: 15923, Time since last communication: 0.000s
I20251026 03:12:08.879056 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:08.879168 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 36 FOLLOWER]: Advancing to term 37
I20251026 03:12:08.879220 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:08.879285 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 36 FOLLOWER]: Advancing to term 37
I20251026 03:12:08.879767 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.879776 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.879914 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 37 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.879952 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 37 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:08.880072 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 37 candidate_status { last_received { term: 36 index: 15924 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.880139 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 37 candidate_status { last_received { term: 36 index: 15924 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:08.880262 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 37 candidate_status { last_received { term: 36 index: 15924 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.880337 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 37: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:08.880072 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 37 candidate_status { last_received { term: 36 index: 15924 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.880396 7766 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 37: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:08.880448 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 37 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 36. Current term is 37. Ops: []
I20251026 03:12:08.880501 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.880539 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:08.880636 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20251026 03:12:08.880615 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20251026 03:12:08.880615 8066 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 36.15924, Next index: 15925, Last known committed idx: 15924, Time since last communication: 0.000s
I20251026 03:12:08.884966 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 37 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 36. Current term is 37. Ops: []
I20251026 03:12:08.885152 8066 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 36.15924, Next index: 15925, Last known committed idx: 15924, Time since last communication: 0.000s
I20251026 03:12:08.889935 8058 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 36 LEADER]: Stepping down as leader of term 36
I20251026 03:12:08.889983 8058 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:08.890057 8058 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15924, Committed index: 15924, Last appended: 36.15924, Last appended by leader: 15924, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.890129 8058 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 36 FOLLOWER]: Advancing to term 37
I20251026 03:12:08.917608 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:08.917696 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 37 FOLLOWER]: Advancing to term 38
I20251026 03:12:08.918413 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.918567 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 38 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:08.918774 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 38 candidate_status { last_received { term: 36 index: 15924 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:08.918774 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 38 candidate_status { last_received { term: 36 index: 15924 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:08.918897 7765 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 37 FOLLOWER]: Advancing to term 38
I20251026 03:12:08.918936 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 37 FOLLOWER]: Advancing to term 38
I20251026 03:12:08.919427 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 38.
I20251026 03:12:08.919530 7765 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 38.
I20251026 03:12:08.919628 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters:
I20251026 03:12:08.919716 8058 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 38 FOLLOWER]: Leader election won for term 38
I20251026 03:12:08.919812 8058 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 38 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:08.919860 8058 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15924, Committed index: 15924, Last appended: 36.15924, Last appended by leader: 15924, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:08.920259 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 36 to 38. New cstate: current_term: 38 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:08.947472 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 38 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 36 index: 15924. Preceding OpId from leader: term: 38 index: 15925. (index mismatch)
I20251026 03:12:08.947733 8058 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15925, Last known committed idx: 15924, Time since last communication: 0.000s
I20251026 03:12:08.956555 7765 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 38 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 36 index: 15924. Preceding OpId from leader: term: 38 index: 15925. (index mismatch)
I20251026 03:12:08.956797 8058 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15925, Last known committed idx: 15924, Time since last communication: 0.000s
I20251026 03:12:09.002133 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:09.002221 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 38 FOLLOWER]: Advancing to term 39
I20251026 03:12:09.002767 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.002893 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 39 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.003090 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 39 candidate_status { last_received { term: 38 index: 15925 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.003090 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 39 candidate_status { last_received { term: 38 index: 15925 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.003334 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.003453 8045 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20251026 03:12:09.007048 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 39 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 38. Current term is 39. Ops: []
I20251026 03:12:09.007256 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 38.15925, Next index: 15926, Last known committed idx: 15925, Time since last communication: 0.000s
I20251026 03:12:09.032159 8065 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 38 LEADER]: Stepping down as leader of term 38
I20251026 03:12:09.032239 8065 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:09.032294 8065 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15925, Committed index: 15925, Last appended: 38.15925, Last appended by leader: 15925, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.032398 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 38 FOLLOWER]: Advancing to term 39
I20251026 03:12:09.038849 8045 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.038925 8045 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 39 FOLLOWER]: Advancing to term 40
I20251026 03:12:09.039520 8045 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.039665 8045 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 40 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.039893 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 40 candidate_status { last_received { term: 38 index: 15925 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.040000 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 39 FOLLOWER]: Advancing to term 40
I20251026 03:12:09.039893 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 40 candidate_status { last_received { term: 38 index: 15925 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.040601 7842 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 40.
I20251026 03:12:09.040773 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040
I20251026 03:12:09.040854 8045 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 40 FOLLOWER]: Leader election won for term 40
I20251026 03:12:09.040917 8045 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 40 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:09.040972 8045 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15925, Committed index: 15925, Last appended: 38.15925, Last appended by leader: 15925, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.041425 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 38 to 40, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 40 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:09.044591 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:09.044637 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 38 FOLLOWER]: Advancing to term 39
I20251026 03:12:09.045120 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.045236 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 39 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.045374 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 39 candidate_status { last_received { term: 38 index: 15925 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.045384 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 39 candidate_status { last_received { term: 38 index: 15925 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.045450 7842 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 39. Current term is 40.
I20251026 03:12:09.045594 7732 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 39 election: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 39. Current term is 40.
I20251026 03:12:09.045630 7732 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 39 election: Cancelling election due to peer responding with higher term
I20251026 03:12:09.045684 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 39 FOLLOWER]: Advancing to term 40
I20251026 03:12:09.046175 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 39. Current term is 40.
I20251026 03:12:09.064563 7842 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 38 index: 15925. Preceding OpId from leader: term: 40 index: 15926. (index mismatch)
I20251026 03:12:09.064814 8045 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15926, Last known committed idx: 15925, Time since last communication: 0.000s
I20251026 03:12:09.075649 7765 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 40 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 38 index: 15925. Preceding OpId from leader: term: 40 index: 15926. (index mismatch)
I20251026 03:12:09.075867 8045 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15926, Last known committed idx: 15925, Time since last communication: 0.000s
I20251026 03:12:09.106742 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:09.106832 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 40 FOLLOWER]: Advancing to term 41
I20251026 03:12:09.107465 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.107580 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 41. Ops: []
I20251026 03:12:09.107611 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 41 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.107743 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 41 candidate_status { last_received { term: 40 index: 15926 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.107882 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 41 candidate_status { last_received { term: 40 index: 15926 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.107954 8067 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.108166 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.108249 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20251026 03:12:09.118945 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:09.119022 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 40 FOLLOWER]: Advancing to term 41
I20251026 03:12:09.119660 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.119825 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 41 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.120002 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 41 candidate_status { last_received { term: 40 index: 15926 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.120059 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 41 candidate_status { last_received { term: 40 index: 15926 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.120098 7765 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 41: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:09.120184 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 41 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 41. Ops: []
I20251026 03:12:09.120283 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.120383 8065 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20251026 03:12:09.120364 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.138554 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 41. Ops: []
I20251026 03:12:09.138793 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.147902 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 41 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 41. Ops: []
I20251026 03:12:09.148162 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
W20251026 03:12:09.150874 7975 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
I20251026 03:12:09.151571 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.151609 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 41 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 41. Ops: [40.15927-40.15927]
I20251026 03:12:09.151623 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 41 FOLLOWER]: Advancing to term 42
I20251026 03:12:09.151798 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.152395 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.152500 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 42. Ops: [40.15927-40.15927]
I20251026 03:12:09.152534 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 42 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.152644 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 42 candidate_status { last_received { term: 40 index: 15926 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.152654 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 42 candidate_status { last_received { term: 40 index: 15926 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.152773 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 41 FOLLOWER]: Advancing to term 42
I20251026 03:12:09.152848 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.153334 7842 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 42.
I20251026 03:12:09.153505 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.153581 7991 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 FOLLOWER]: Leader election won for term 42
I20251026 03:12:09.153673 7991 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.153731 7991 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15926, Committed index: 15926, Last appended: 40.15926, Last appended by leader: 15926, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.154146 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 40 to 42, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 42 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:09.174288 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 42 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 40. Current term is 42. Ops: [40.15927-40.15927]
I20251026 03:12:09.174516 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 40.15926, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.180610 7917 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 40 LEADER]: Stepping down as leader of term 40
I20251026 03:12:09.180665 7917 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:09.180716 7917 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15926, Committed index: 15926, Last appended: 40.15927, Last appended by leader: 15927, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.180795 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 40 FOLLOWER]: Advancing to term 42
I20251026 03:12:09.181427 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 42 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 40 index: 15927. Preceding OpId from leader: term: 42 index: 15927. (term mismatch)
I20251026 03:12:09.181465 7917 pending_rounds.cc:85] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting all ops after (but not including) 15926
I20251026 03:12:09.181476 7917 pending_rounds.cc:107] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 40.15927
I20251026 03:12:09.181674 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.183034 7842 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 42 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 40 index: 15926. Preceding OpId from leader: term: 42 index: 15928. (index mismatch)
I20251026 03:12:09.183350 8095 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15927, Last known committed idx: 15926, Time since last communication: 0.000s
I20251026 03:12:09.214134 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 42 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.214213 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 42 FOLLOWER]: Advancing to term 43
I20251026 03:12:09.214867 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.214990 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 43 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.215131 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 43 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.215175 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 43 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.215386 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.215483 8067 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20251026 03:12:09.218511 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 42. Current term is 43. Ops: []
I20251026 03:12:09.218715 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.241369 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 42 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.241433 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 42 FOLLOWER]: Advancing to term 43
I20251026 03:12:09.242038 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.242178 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 43 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.242354 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 43 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.242430 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 43: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:09.242359 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 43 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.242573 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.242662 8065 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20251026 03:12:09.242729 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 43 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 42. Current term is 43. Ops: []
I20251026 03:12:09.242928 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.243018 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 42. Current term is 43. Ops: []
I20251026 03:12:09.243266 8075 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.251161 7991 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 LEADER]: Stepping down as leader of term 42
I20251026 03:12:09.251209 7991 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.251253 7991 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15928, Committed index: 15928, Last appended: 42.15928, Last appended by leader: 15928, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.251343 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 42 FOLLOWER]: Advancing to term 43
I20251026 03:12:09.251870 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.251924 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 43 FOLLOWER]: Advancing to term 44
I20251026 03:12:09.252367 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 44 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.252450 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 44 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.252609 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 44 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.252632 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 44 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.252681 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 43 FOLLOWER]: Advancing to term 44
I20251026 03:12:09.252686 7765 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 43 FOLLOWER]: Advancing to term 44
I20251026 03:12:09.253165 7765 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 44.
I20251026 03:12:09.253181 7842 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 44.
I20251026 03:12:09.253330 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters:
I20251026 03:12:09.253409 8067 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 44 FOLLOWER]: Leader election won for term 44
I20251026 03:12:09.253454 8067 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 44 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:09.253504 8067 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15928, Committed index: 15928, Last appended: 42.15928, Last appended by leader: 15928, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.253922 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 42 to 44, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 44 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
W20251026 03:12:09.272259 8053 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 44 is higher.
I20251026 03:12:09.277084 7842 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 44 FOLLOWER]: Refusing update from remote peer 360eb6050b7244d58989794aea84cc01: Log matching property violated. Preceding OpId in replica: term: 42 index: 15928. Preceding OpId from leader: term: 44 index: 15929. (index mismatch)
I20251026 03:12:09.277321 8067 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.284621 8095 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 44 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.284675 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 44 FOLLOWER]: Advancing to term 45
I20251026 03:12:09.285291 8095 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.285418 8095 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 45 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.285419 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 45 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 45. Ops: []
I20251026 03:12:09.285513 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 45 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.285521 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 45 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.285686 8067 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.285882 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.285969 8095 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
W20251026 03:12:09.297617 8053 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 44 is higher.
I20251026 03:12:09.322242 8095 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 45 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.322320 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 45 FOLLOWER]: Advancing to term 46
I20251026 03:12:09.322914 8095 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.323027 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 46. Ops: [44.15929-44.15929]
I20251026 03:12:09.323062 8095 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 46 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.323197 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 46 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.323243 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 46 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.323278 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.323410 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.323489 8095 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
W20251026 03:12:09.335855 8053 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 44 is higher.
W20251026 03:12:09.335987 8053 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 42 current term: 44 is higher.
I20251026 03:12:09.358238 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 46. Ops: [44.15929-44.15929]
I20251026 03:12:09.358510 8067 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.360471 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 44 FOLLOWER]: Starting leader election (detected failure of leader 360eb6050b7244d58989794aea84cc01)
I20251026 03:12:09.360509 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 44 FOLLOWER]: Advancing to term 45
I20251026 03:12:09.361198 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 45 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.361318 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 45 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.361435 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 45 candidate_status { last_received { term: 44 index: 15929 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.361441 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 45 candidate_status { last_received { term: 44 index: 15929 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.361539 7765 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 45. Current term is 46.
I20251026 03:12:09.361711 7808 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 45 election: Vote denied by peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 45. Current term is 46.
I20251026 03:12:09.361743 7808 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 45 election: Cancelling election due to peer responding with higher term
I20251026 03:12:09.361822 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 45 FOLLOWER]: Advancing to term 46
I20251026 03:12:09.362370 8065 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 46 FOLLOWER]: Leader election lost for term 45. Reason: Vote denied by peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a for earlier term 45. Current term is 46.
I20251026 03:12:09.365947 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 46 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 46. Ops: []
I20251026 03:12:09.366117 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 44.15929, Next index: 15930, Last known committed idx: 15929, Time since last communication: 0.000s
W20251026 03:12:09.373224 8053 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 42 current term: 44 is higher.
I20251026 03:12:09.373584 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 46. Ops: [44.15929-44.15929]
I20251026 03:12:09.373790 8067 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.383952 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.384008 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 46 FOLLOWER]: Advancing to term 47
I20251026 03:12:09.384503 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 47 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.384605 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 47 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.384732 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 47 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.384805 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 47 candidate_status { last_received { term: 42 index: 15928 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.384873 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 46 FOLLOWER]: Advancing to term 47
I20251026 03:12:09.385366 7842 raft_consensus.cc:2410] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 for term 47 because replica has last-logged OpId of term: 44 index: 15929, which is greater than that of the candidate, which has last-logged OpId of term: 42 index: 15928.
I20251026 03:12:09.385527 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.385598 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
I20251026 03:12:09.388018 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 47 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 47. Ops: []
I20251026 03:12:09.388226 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 44.15929, Next index: 15930, Last known committed idx: 15929, Time since last communication: 0.000s
I20251026 03:12:09.406823 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 47 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.406885 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 47 FOLLOWER]: Advancing to term 48
I20251026 03:12:09.407428 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.407546 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 48 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.407712 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 48 candidate_status { last_received { term: 44 index: 15929 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.407721 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 48 candidate_status { last_received { term: 44 index: 15929 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.407809 7765 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 47 FOLLOWER]: Advancing to term 48
I20251026 03:12:09.408385 7765 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 48.
I20251026 03:12:09.408569 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.408661 8065 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 FOLLOWER]: Leader election won for term 48
I20251026 03:12:09.408716 8065 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:09.408761 8065 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15929, Committed index: 15929, Last appended: 44.15929, Last appended by leader: 15929, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.409212 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 48 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 48. Ops: [44.15929-44.15929]
I20251026 03:12:09.409228 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 44 to 48, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 48 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:09.409394 8045 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 42.15928, Next index: 15929, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.410075 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 LEADER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 44. Current term is 48. Ops: []
I20251026 03:12:09.410226 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 44.15929, Next index: 15930, Last known committed idx: 15929, Time since last communication: 0.000s
I20251026 03:12:09.410284 8053 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer 5340c78feac8406882e860a2bed2ec2a log is divergent from this leader: its last log entry 48.15930 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15929
I20251026 03:12:09.430644 8067 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 44 LEADER]: Stepping down as leader of term 44
I20251026 03:12:09.430706 8067 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:09.430765 8067 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15929, Committed index: 15929, Last appended: 44.15929, Last appended by leader: 15929, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.430855 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 44 FOLLOWER]: Advancing to term 45
I20251026 03:12:09.438410 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 45 FOLLOWER]: Advancing to term 48
I20251026 03:12:09.438917 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 48 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 44 index: 15929. Preceding OpId from leader: term: 48 index: 15930. (index mismatch)
I20251026 03:12:09.439168 8065 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15930, Last known committed idx: 15929, Time since last communication: 0.000s
I20251026 03:12:09.442303 7765 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 48 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 42 index: 15928. Preceding OpId from leader: term: 48 index: 15930. (index mismatch)
I20251026 03:12:09.442485 8065 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15930, Last known committed idx: 15928, Time since last communication: 0.000s
I20251026 03:12:09.473742 8095 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:09.473848 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 48 FOLLOWER]: Advancing to term 49
I20251026 03:12:09.474439 8095 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.474550 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 48. Current term is 49. Ops: []
I20251026 03:12:09.474570 8095 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 49 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.474670 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 49 candidate_status { last_received { term: 48 index: 15930 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.474686 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 49 candidate_status { last_received { term: 48 index: 15930 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.474740 8065 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 48.15930, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.474889 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.474977 8095 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20251026 03:12:09.497115 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 48. Current term is 49. Ops: []
I20251026 03:12:09.497381 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 48.15930, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.500206 8053 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 48 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:09.500257 8053 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 48 FOLLOWER]: Advancing to term 49
I20251026 03:12:09.500805 8053 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 49 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.500918 8053 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 49 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.501096 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 49 candidate_status { last_received { term: 48 index: 15930 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.501096 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 49 candidate_status { last_received { term: 48 index: 15930 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.501202 7765 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 49: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:09.501314 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 49 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.501387 8053 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 49 FOLLOWER]: Leader election lost for term 49. Reason: could not achieve majority
I20251026 03:12:09.501602 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 49 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 48. Current term is 49. Ops: []
I20251026 03:12:09.501753 8065 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 48.15930, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.519264 8095 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.519317 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 49 FOLLOWER]: Advancing to term 50
I20251026 03:12:09.519950 8095 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.520077 8095 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 50 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.520183 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 50 candidate_status { last_received { term: 48 index: 15930 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.520213 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 50 candidate_status { last_received { term: 48 index: 15930 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.520269 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 49 FOLLOWER]: Advancing to term 50
I20251026 03:12:09.520707 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 50.
I20251026 03:12:09.520871 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters: 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.520951 8095 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 FOLLOWER]: Leader election won for term 50
I20251026 03:12:09.520992 8095 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.521047 8095 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15930, Committed index: 15930, Last appended: 48.15930, Last appended by leader: 15930, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.521421 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 48 to 50, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 50 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:09.530936 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 50 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 48. Current term is 50. Ops: []
I20251026 03:12:09.531116 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 48.15930, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.531193 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 LEADER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 48. Current term is 50. Ops: []
I20251026 03:12:09.531389 8065 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 48.15930, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.531453 8065 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer 2a26a9c2ccd44b42a5d55fe3c9607040 log is divergent from this leader: its last log entry 50.15931 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15930
I20251026 03:12:09.546741 7842 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 LEADER]: Stepping down as leader of term 48
I20251026 03:12:09.546804 7842 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:09.546862 7842 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15930, Committed index: 15930, Last appended: 48.15930, Last appended by leader: 15930, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.546936 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 48 FOLLOWER]: Advancing to term 50
I20251026 03:12:09.547533 7842 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 50 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 48 index: 15930. Preceding OpId from leader: term: 50 index: 15931. (index mismatch)
I20251026 03:12:09.547574 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 50 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 48 index: 15930. Preceding OpId from leader: term: 50 index: 15931. (index mismatch)
I20251026 03:12:09.547731 8095 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.547799 8075 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15931, Last known committed idx: 15930, Time since last communication: 0.000s
I20251026 03:12:09.605032 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 50 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.605113 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 50 FOLLOWER]: Advancing to term 51
I20251026 03:12:09.605662 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.605810 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 51 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.605931 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 51 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.605947 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 51 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.606159 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 50 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.606181 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.606207 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 50 FOLLOWER]: Advancing to term 51
I20251026 03:12:09.606254 8067 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20251026 03:12:09.606724 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 51 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.606839 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 51 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.606995 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 51 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.607051 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 51 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.607084 7842 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 51 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 50. Current term is 51. Ops: []
I20251026 03:12:09.607120 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 51: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:09.607262 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 50.15931, Next index: 15932, Last known committed idx: 15931, Time since last communication: 0.000s
I20251026 03:12:09.607314 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.607378 8065 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20251026 03:12:09.609429 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 51 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 50. Current term is 51. Ops: []
I20251026 03:12:09.609615 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 50.15931, Next index: 15932, Last known committed idx: 15931, Time since last communication: 0.000s
I20251026 03:12:09.613456 7991 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 LEADER]: Stepping down as leader of term 50
I20251026 03:12:09.613499 7991 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.613536 7991 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15931, Committed index: 15931, Last appended: 50.15931, Last appended by leader: 15931, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.613591 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 50 FOLLOWER]: Advancing to term 51
I20251026 03:12:09.643019 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.643101 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 51 FOLLOWER]: Advancing to term 52
I20251026 03:12:09.643647 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 51 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.643700 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 51 FOLLOWER]: Advancing to term 52
I20251026 03:12:09.643672 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.643865 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 52 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.644028 7842 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 52 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.644028 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 52 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.644119 7842 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 51 FOLLOWER]: Advancing to term 52
I20251026 03:12:09.644471 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 52 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.644591 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 52 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.644584 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in current term 52: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:09.644577 7842 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 52.
I20251026 03:12:09.644747 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 52 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.644810 7765 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 52: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:09.644784 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 52 candidate_status { last_received { term: 50 index: 15931 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.644850 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 52: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:09.644879 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.644984 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.645010 7991 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 FOLLOWER]: Leader election won for term 52
I20251026 03:12:09.645062 8067 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20251026 03:12:09.645087 7991 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.645125 7991 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15931, Committed index: 15931, Last appended: 50.15931, Last appended by leader: 15931, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.645515 7656 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 50 to 52. New cstate: current_term: 52 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:09.672029 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 52 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 50 index: 15931. Preceding OpId from leader: term: 52 index: 15932. (index mismatch)
I20251026 03:12:09.672291 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15932, Last known committed idx: 15931, Time since last communication: 0.000s
I20251026 03:12:09.677851 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 52 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 50 index: 15931. Preceding OpId from leader: term: 52 index: 15932. (index mismatch)
I20251026 03:12:09.678104 7991 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15932, Last known committed idx: 15931, Time since last communication: 0.000s
I20251026 03:12:09.709358 8066 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 52 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.709439 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 52 FOLLOWER]: Advancing to term 53
I20251026 03:12:09.710072 8066 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.710196 8066 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 53 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.710366 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 53 candidate_status { last_received { term: 52 index: 15932 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.710354 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 53 candidate_status { last_received { term: 52 index: 15932 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.710595 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.710676 8066 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20251026 03:12:09.712841 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 52. Current term is 53. Ops: []
I20251026 03:12:09.713032 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 52.15932, Next index: 15933, Last known committed idx: 15932, Time since last communication: 0.000s
I20251026 03:12:09.734012 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 52 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.734077 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 52 FOLLOWER]: Advancing to term 53
I20251026 03:12:09.734671 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 53 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.734793 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 53 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.734959 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 53 candidate_status { last_received { term: 52 index: 15932 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.735034 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 53: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:09.734997 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 53 candidate_status { last_received { term: 52 index: 15932 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.735265 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.735344 8067 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20251026 03:12:09.736392 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 52. Current term is 53. Ops: []
I20251026 03:12:09.736544 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 52.15932, Next index: 15933, Last known committed idx: 15932, Time since last communication: 0.000s
I20251026 03:12:09.736974 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 53 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 52. Current term is 53. Ops: []
I20251026 03:12:09.737126 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 52.15932, Next index: 15933, Last known committed idx: 15932, Time since last communication: 0.000s
I20251026 03:12:09.754988 8066 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.755069 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 53 FOLLOWER]: Advancing to term 54
I20251026 03:12:09.755707 8066 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 54 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.755867 8066 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 54 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.756042 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 54 candidate_status { last_received { term: 52 index: 15932 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.756042 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 54 candidate_status { last_received { term: 52 index: 15932 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.756142 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 53 FOLLOWER]: Advancing to term 54
I20251026 03:12:09.756662 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 54.
I20251026 03:12:09.756836 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040
I20251026 03:12:09.756919 8066 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 54 FOLLOWER]: Leader election won for term 54
I20251026 03:12:09.756976 8066 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 54 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:09.757035 8066 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15932, Committed index: 15932, Last appended: 52.15932, Last appended by leader: 15932, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.757398 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a reported cstate change: term changed from 52 to 54, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 54 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:09.759545 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 54 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 52. Current term is 54. Ops: []
I20251026 03:12:09.759732 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 52.15932, Next index: 15933, Last known committed idx: 15932, Time since last communication: 0.000s
I20251026 03:12:09.770449 8075 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 LEADER]: Stepping down as leader of term 52
I20251026 03:12:09.770501 8075 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.770556 8075 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15932, Committed index: 15932, Last appended: 52.15932, Last appended by leader: 15932, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.770642 8075 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 52 FOLLOWER]: Advancing to term 53
I20251026 03:12:09.785620 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 54 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 52 index: 15932. Preceding OpId from leader: term: 54 index: 15933. (index mismatch)
I20251026 03:12:09.785843 8066 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15933, Last known committed idx: 15932, Time since last communication: 0.000s
I20251026 03:12:09.791661 7765 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 53 FOLLOWER]: Advancing to term 54
I20251026 03:12:09.792272 7765 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 54 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 52 index: 15932. Preceding OpId from leader: term: 54 index: 15933. (index mismatch)
I20251026 03:12:09.792472 8066 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15933, Last known committed idx: 15932, Time since last communication: 0.000s
I20251026 03:12:09.851506 7991 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 54 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:09.851591 7991 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 54 FOLLOWER]: Advancing to term 55
I20251026 03:12:09.852151 7991 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.852349 7991 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 55 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.852432 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 55 candidate_status { last_received { term: 54 index: 15933 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.852425 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 55 candidate_status { last_received { term: 54 index: 15933 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.852687 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:09.852784 7991 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20251026 03:12:09.857127 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 55 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 54. Current term is 55. Ops: []
I20251026 03:12:09.857313 8065 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 54.15933, Next index: 15934, Last known committed idx: 15933, Time since last communication: 0.000s
I20251026 03:12:09.883900 8066 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 54 LEADER]: Stepping down as leader of term 54
I20251026 03:12:09.883975 8066 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:09.884023 8066 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15933, Committed index: 15933, Last appended: 54.15933, Last appended by leader: 15933, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.884120 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 54 FOLLOWER]: Advancing to term 55
I20251026 03:12:09.889498 8075 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 55 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:09.889561 8075 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 55 FOLLOWER]: Advancing to term 56
I20251026 03:12:09.890187 8075 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 56 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.890312 8075 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 56 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:09.890489 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 56 candidate_status { last_received { term: 54 index: 15933 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.890511 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 56 candidate_status { last_received { term: 54 index: 15933 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.890596 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 55 FOLLOWER]: Advancing to term 56
I20251026 03:12:09.891084 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 56.
I20251026 03:12:09.891259 7732 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.891340 8075 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 56 FOLLOWER]: Leader election won for term 56
I20251026 03:12:09.891386 8075 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 56 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.891436 8075 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15933, Committed index: 15933, Last appended: 54.15933, Last appended by leader: 15933, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.891844 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 54 to 56, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 56 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:09.895638 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 54 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:09.895679 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 54 FOLLOWER]: Advancing to term 55
I20251026 03:12:09.896107 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 55 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.896211 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 55 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.896380 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 55 candidate_status { last_received { term: 54 index: 15933 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:09.896409 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 55 candidate_status { last_received { term: 54 index: 15933 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.896451 7841 raft_consensus.cc:2368] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 55. Current term is 56.
I20251026 03:12:09.896569 7883 leader_election.cc:400] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 55 election: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 55. Current term is 56.
I20251026 03:12:09.896601 7883 leader_election.cc:403] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 55 election: Cancelling election due to peer responding with higher term
I20251026 03:12:09.896663 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 55 FOLLOWER]: Advancing to term 56
I20251026 03:12:09.897143 8067 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 56 FOLLOWER]: Leader election lost for term 55. Reason: Vote denied by peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837) with higher term. Message: Invalid argument: T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 for earlier term 55. Current term is 56.
I20251026 03:12:09.915344 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 54 index: 15933. Preceding OpId from leader: term: 56 index: 15934. (index mismatch)
I20251026 03:12:09.915607 8075 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15934, Last known committed idx: 15933, Time since last communication: 0.000s
I20251026 03:12:09.925113 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 56 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 54 index: 15933. Preceding OpId from leader: term: 56 index: 15934. (index mismatch)
I20251026 03:12:09.925308 8075 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15934, Last known committed idx: 15933, Time since last communication: 0.000s
I20251026 03:12:09.974524 8066 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:09.974604 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 56 FOLLOWER]: Advancing to term 57
I20251026 03:12:09.975193 8066 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.975301 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 57 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 56. Current term is 57. Ops: []
I20251026 03:12:09.975427 8066 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 57 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:09.975466 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 57 candidate_status { last_received { term: 56 index: 15934 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:09.975482 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 57 candidate_status { last_received { term: 56 index: 15934 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:09.975483 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 56.15934, Next index: 15935, Last known committed idx: 15934, Time since last communication: 0.000s
I20251026 03:12:09.975667 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:09.975742 8066 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20251026 03:12:09.996798 8095 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 56 LEADER]: Stepping down as leader of term 56
I20251026 03:12:09.996863 8095 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:09.996914 8095 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15934, Committed index: 15934, Last appended: 56.15934, Last appended by leader: 15934, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:09.996999 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 56 FOLLOWER]: Advancing to term 57
I20251026 03:12:10.010069 8067 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 56 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:10.010139 8067 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 56 FOLLOWER]: Advancing to term 57
I20251026 03:12:10.010669 8067 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 57 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.010797 8067 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 57 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.010979 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 57 candidate_status { last_received { term: 56 index: 15934 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:10.010996 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 57 candidate_status { last_received { term: 56 index: 15934 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.011041 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 57: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:10.011595 7765 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 57.
I20251026 03:12:10.011752 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters: 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:10.011838 8067 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 57 FOLLOWER]: Leader election won for term 57
I20251026 03:12:10.011893 8067 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 57 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:10.011940 8067 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15934, Committed index: 15934, Last appended: 56.15934, Last appended by leader: 15934, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.012301 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 56 to 57, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 57 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:10.019126 8066 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:10.019196 8066 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 57 FOLLOWER]: Advancing to term 58
I20251026 03:12:10.019680 8066 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.019798 8066 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 58 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.019950 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 58 candidate_status { last_received { term: 56 index: 15934 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.019969 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 58 candidate_status { last_received { term: 56 index: 15934 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:10.020023 7765 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 57 FOLLOWER]: Advancing to term 58
I20251026 03:12:10.020629 7765 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5340c78feac8406882e860a2bed2ec2a in term 58.
I20251026 03:12:10.020787 7808 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a; no voters: 360eb6050b7244d58989794aea84cc01
I20251026 03:12:10.020876 8066 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 FOLLOWER]: Leader election won for term 58
I20251026 03:12:10.020967 8066 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 LEADER]: Becoming Leader. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:10.021051 8066 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15934, Committed index: 15934, Last appended: 56.15934, Last appended by leader: 15934, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.042618 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 LEADER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 57. Current term is 58. Ops: []
I20251026 03:12:10.042618 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 58 FOLLOWER]: Rejecting Update request from peer 360eb6050b7244d58989794aea84cc01 for earlier term 57. Current term is 58. Ops: []
I20251026 03:12:10.042872 8053 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15935, Last known committed idx: 15934, Time since last communication: 0.000s
I20251026 03:12:10.043030 8053 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer 5340c78feac8406882e860a2bed2ec2a log is divergent from this leader: its last log entry 58.15935 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15934
I20251026 03:12:10.043080 8067 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15935, Last known committed idx: 15934, Time since last communication: 0.000s
I20251026 03:12:10.046198 7765 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 58 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 56 index: 15934. Preceding OpId from leader: term: 58 index: 15935. (index mismatch)
I20251026 03:12:10.046402 8066 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15935, Last known committed idx: 15934, Time since last communication: 0.000s
I20251026 03:12:10.055145 8102 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 57 LEADER]: Stepping down as leader of term 57
I20251026 03:12:10.055188 8102 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:10.055243 8102 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15934, Committed index: 15934, Last appended: 57.15935, Last appended by leader: 15935, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.055317 8102 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 57 FOLLOWER]: Advancing to term 58
I20251026 03:12:10.055963 7656 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 57 to 58, leader changed from 360eb6050b7244d58989794aea84cc01 (127.30.128.67) to 5340c78feac8406882e860a2bed2ec2a (127.30.128.66). New cstate: current_term: 58 leader_uuid: "5340c78feac8406882e860a2bed2ec2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } } }
I20251026 03:12:10.058264 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 58 FOLLOWER]: Refusing update from remote peer 5340c78feac8406882e860a2bed2ec2a: Log matching property violated. Preceding OpId in replica: term: 57 index: 15935. Preceding OpId from leader: term: 58 index: 15935. (term mismatch)
I20251026 03:12:10.058305 7917 pending_rounds.cc:85] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting all ops after (but not including) 15934
I20251026 03:12:10.058318 7917 pending_rounds.cc:107] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting uncommitted NO_OP operation due to leader change: 57.15935
I20251026 03:12:10.058331 7917 raft_consensus.cc:2889] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 03:12:10.058499 8066 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15935, Last known committed idx: 15934, Time since last communication: 0.000s
I20251026 03:12:10.077555 8095 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 58 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:10.077608 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 58 FOLLOWER]: Advancing to term 59
I20251026 03:12:10.078210 8095 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.078321 8095 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 59 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:10.078496 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 59 candidate_status { last_received { term: 58 index: 15935 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:10.078500 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 59 candidate_status { last_received { term: 58 index: 15935 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:10.078723 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040; no voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:10.078852 8095 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I20251026 03:12:10.082002 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 58. Current term is 59. Ops: []
I20251026 03:12:10.082167 8065 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 58.15935, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.089632 8102 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 58 FOLLOWER]: Starting leader election (detected failure of leader 5340c78feac8406882e860a2bed2ec2a)
I20251026 03:12:10.089679 8102 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 58 FOLLOWER]: Advancing to term 59
I20251026 03:12:10.090240 8102 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 59 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.090351 8102 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 59 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.090503 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 59 candidate_status { last_received { term: 58 index: 15935 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:10.090541 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 59 candidate_status { last_received { term: 58 index: 15935 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.090611 7765 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 59: Already voted for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in this term.
I20251026 03:12:10.090737 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 59 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:10.090824 8102 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 59 FOLLOWER]: Leader election lost for term 59. Reason: could not achieve majority
I20251026 03:12:10.091905 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 59 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 58. Current term is 59. Ops: []
I20251026 03:12:10.092089 8066 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 58.15935, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.104920 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 58. Current term is 59. Ops: []
I20251026 03:12:10.105157 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 58.15935, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.116364 8095 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:10.116421 8095 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 59 FOLLOWER]: Advancing to term 60
I20251026 03:12:10.116968 8095 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.117101 8095 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 60 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837)
I20251026 03:12:10.117249 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 60 candidate_status { last_received { term: 58 index: 15935 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:10.117275 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" candidate_term: 60 candidate_status { last_received { term: 58 index: 15935 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:10.117329 7917 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 59 FOLLOWER]: Advancing to term 60
I20251026 03:12:10.117859 7917 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 60 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a26a9c2ccd44b42a5d55fe3c9607040 in term 60.
I20251026 03:12:10.118055 7730 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [CANDIDATE]: Term 60 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01; no voters: 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:10.118158 8095 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 FOLLOWER]: Leader election won for term 60
I20251026 03:12:10.118217 8095 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 LEADER]: Becoming Leader. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:10.118275 8095 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15935, Committed index: 15935, Last appended: 58.15935, Last appended by leader: 15935, Current term: 60, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.118671 7660 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 reported cstate change: term changed from 58 to 60, leader changed from 5340c78feac8406882e860a2bed2ec2a (127.30.128.66) to 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65). New cstate: current_term: 60 leader_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:10.121708 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 60 FOLLOWER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 58. Current term is 60. Ops: []
I20251026 03:12:10.121879 8058 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 58.15935, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.132265 7765 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 LEADER]: Rejecting Update request from peer 5340c78feac8406882e860a2bed2ec2a for earlier term 58. Current term is 60. Ops: []
I20251026 03:12:10.132473 8065 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 }, Status: INVALID_TERM, Last received: 58.15935, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.132552 8065 consensus_queue.cc:1243] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [LEADER]: Peer 2a26a9c2ccd44b42a5d55fe3c9607040 log is divergent from this leader: its last log entry 60.15936 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15935
I20251026 03:12:10.140889 7917 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 60 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 58 index: 15935. Preceding OpId from leader: term: 60 index: 15936. (index mismatch)
I20251026 03:12:10.141073 8095 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.145978 7841 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 LEADER]: Stepping down as leader of term 58
I20251026 03:12:10.146023 7841 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 LEADER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Running, Role: LEADER
I20251026 03:12:10.146067 7841 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15935, Committed index: 15935, Last appended: 58.15935, Last appended by leader: 15935, Current term: 58, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.146158 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 58 FOLLOWER]: Advancing to term 60
I20251026 03:12:10.146688 7841 raft_consensus.cc:1275] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 60 FOLLOWER]: Refusing update from remote peer 2a26a9c2ccd44b42a5d55fe3c9607040: Log matching property violated. Preceding OpId in replica: term: 58 index: 15935. Preceding OpId from leader: term: 60 index: 15936. (index mismatch)
I20251026 03:12:10.146903 8095 consensus_queue.cc:1048] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15936, Last known committed idx: 15935, Time since last communication: 0.000s
I20251026 03:12:10.172408 8102 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 60 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:10.172470 8102 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 60 FOLLOWER]: Advancing to term 61
I20251026 03:12:10.173009 8102 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.173143 8102 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 61 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.173269 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 61 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:10.173321 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 61 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.173467 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:10.173560 8102 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20251026 03:12:10.177426 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 60 FOLLOWER]: Starting leader election (detected failure of leader 2a26a9c2ccd44b42a5d55fe3c9607040)
I20251026 03:12:10.177476 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 60 FOLLOWER]: Advancing to term 61
I20251026 03:12:10.177986 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 60. Current term is 61. Ops: []
I20251026 03:12:10.178155 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 61 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.178213 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 60.15936, Next index: 15937, Last known committed idx: 15936, Time since last communication: 0.000s
I20251026 03:12:10.178287 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 61 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.178417 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 61 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.178447 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 61 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:10.178495 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 61: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:10.178642 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 61 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:10.178723 8065 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 61 FOLLOWER]: Leader election lost for term 61. Reason: could not achieve majority
I20251026 03:12:10.184090 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 61 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 60. Current term is 61. Ops: []
I20251026 03:12:10.184310 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 60.15936, Next index: 15937, Last known committed idx: 15936, Time since last communication: 0.000s
I20251026 03:12:10.208081 7917 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 60. Current term is 61. Ops: []
I20251026 03:12:10.208348 8095 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 }, Status: INVALID_TERM, Last received: 60.15936, Next index: 15937, Last known committed idx: 15936, Time since last communication: 0.000s
I20251026 03:12:10.213177 8102 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:10.213238 8065 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 61 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:10.213248 8102 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 61 FOLLOWER]: Advancing to term 62
I20251026 03:12:10.213313 8065 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 61 FOLLOWER]: Advancing to term 62
I20251026 03:12:10.213950 8065 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.213951 8102 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 62 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.214056 7841 raft_consensus.cc:1240] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 62 FOLLOWER]: Rejecting Update request from peer 2a26a9c2ccd44b42a5d55fe3c9607040 for earlier term 60. Current term is 62. Ops: []
I20251026 03:12:10.214114 8102 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 62 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.214222 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 62 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
I20251026 03:12:10.214243 7766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 62 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.214255 7991 consensus_queue.cc:1059] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 }, Status: INVALID_TERM, Last received: 60.15936, Next index: 15937, Last known committed idx: 15936, Time since last communication: 0.000s
I20251026 03:12:10.214289 7841 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 360eb6050b7244d58989794aea84cc01 in current term 62: Already voted for candidate 5340c78feac8406882e860a2bed2ec2a in this term.
I20251026 03:12:10.214112 8065 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 62 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.214222 7765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 62 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
I20251026 03:12:10.214390 7917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "5340c78feac8406882e860a2bed2ec2a" candidate_term: 62 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "360eb6050b7244d58989794aea84cc01"
I20251026 03:12:10.214439 7917 raft_consensus.cc:2393] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 62 FOLLOWER]: Leader election vote request: Denying vote to candidate 5340c78feac8406882e860a2bed2ec2a in current term 62: Already voted for candidate 360eb6050b7244d58989794aea84cc01 in this term.
I20251026 03:12:10.214529 7806 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:10.214552 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 62 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:10.214612 8065 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20251026 03:12:10.214620 8102 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 62 FOLLOWER]: Leader election lost for term 62. Reason: could not achieve majority
I20251026 03:12:10.217173 8075 raft_consensus.cc:3055] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 LEADER]: Stepping down as leader of term 60
I20251026 03:12:10.217208 8075 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 LEADER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Running, Role: LEADER
I20251026 03:12:10.217267 8075 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15936, Committed index: 15936, Last appended: 60.15936, Last appended by leader: 15936, Current term: 60, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.217334 8075 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 60 FOLLOWER]: Advancing to term 61
W20251026 03:12:10.234513 7975 lock_manager.cc:494] Txn 3 has not acquired the partition lock after 1000ms
I20251026 03:12:10.239032 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:12:10.241174 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:10.241264 31233 tablet_replica.cc:333] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: stopping tablet replica
I20251026 03:12:10.241315 31233 raft_consensus.cc:2243] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 61 FOLLOWER]: Raft consensus shutting down.
W20251026 03:12:10.241703 8095 raft_consensus.cc:670] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping [suppressed 1 similar messages]
I20251026 03:12:10.274366 8102 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 62 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:10.274451 8102 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 62 FOLLOWER]: Advancing to term 63
I20251026 03:12:10.275092 8102 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.275249 8102 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 63 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:10.275465 7841 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64628560ddff4d0b94db2b5502de12e5" candidate_uuid: "360eb6050b7244d58989794aea84cc01" candidate_term: 63 candidate_status { last_received { term: 60 index: 15936 } } ignore_live_leader: false dest_uuid: "5340c78feac8406882e860a2bed2ec2a"
W20251026 03:12:10.275534 7883 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 03:12:10.275550 7841 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 62 FOLLOWER]: Advancing to term 63
W20251026 03:12:10.275879 7883 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 63 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 03:12:10.276139 7841 raft_consensus.cc:2468] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 360eb6050b7244d58989794aea84cc01 in term 63.
I20251026 03:12:10.276307 7883 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 63 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 360eb6050b7244d58989794aea84cc01, 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040
I20251026 03:12:10.276389 8102 raft_consensus.cc:2804] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Leader election won for term 63
I20251026 03:12:10.276450 8102 raft_consensus.cc:697] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 LEADER]: Becoming Leader. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Running, Role: LEADER
I20251026 03:12:10.276505 8102 consensus_queue.cc:237] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15936, Committed index: 15936, Last appended: 60.15936, Last appended by leader: 15936, Current term: 63, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:10.276894 7655 catalog_manager.cc:5649] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 reported cstate change: term changed from 60 to 63, leader changed from 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65) to 360eb6050b7244d58989794aea84cc01 (127.30.128.67). New cstate: current_term: 63 leader_uuid: "360eb6050b7244d58989794aea84cc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:10.279033 31233 raft_consensus.cc:2272] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 61 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:10.293406 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:12:10.294162 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:12:10.296516 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:10.296622 31233 tablet_replica.cc:333] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: stopping tablet replica
I20251026 03:12:10.296681 31233 raft_consensus.cc:2243] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:10.296715 31233 raft_consensus.cc:2272] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Raft consensus is shut down!
W20251026 03:12:10.301718 7883 consensus_peers.cc:597] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 -> Peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Couldn't send request to peer 5340c78feac8406882e860a2bed2ec2a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251026 03:12:10.306041 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:12:10.306788 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:12:10.309096 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:10.309180 31233 tablet_replica.cc:333] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: stopping tablet replica
I20251026 03:12:10.309258 31233 raft_consensus.cc:2243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 LEADER]: Raft consensus shutting down.
I20251026 03:12:10.309319 31233 pending_rounds.cc:70] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Trying to abort 1 pending ops.
I20251026 03:12:10.309345 31233 pending_rounds.cc:77] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting op as it isn't in flight: id { term: 63 index: 15937 } timestamp: 7214892360813027328 op_type: NO_OP noop_request { }
I20251026 03:12:10.309393 31233 raft_consensus.cc:2889] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:12:10.309425 31233 raft_consensus.cc:2272] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:10.321394 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:12:10.322149 31233 master.cc:561] Master@127.30.128.126:45221 shutting down...
I20251026 03:12:10.324427 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:12:10.324487 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:10.324522 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: stopping tablet replica
I20251026 03:12:10.335945 31233 master.cc:583] Master@127.30.128.126:45221 shutdown complete.
I20251026 03:12:10.337127 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:10.338249 8219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:10.338255 8218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:10.338371 8221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.338409 31233 server_base.cc:1047] running on GCE node
I20251026 03:12:10.338521 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:10.338553 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:10.338569 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448330338570 us; error 0 us; skew 500 ppm
I20251026 03:12:10.339104 31233 webserver.cc:492] Webserver started at http://127.30.128.126:33009/ using document root <none> and password file <none>
I20251026 03:12:10.339169 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:10.339193 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:12:10.339654 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.340011 8226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.340118 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.340167 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "46176127b3ae434bbeba5dfb6dd21469"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:12:10.340205 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:10.359311 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:10.359453 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:10.361651 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:45221
I20251026 03:12:10.367065 8278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:45221 every 8 connection(s)
I20251026 03:12:10.367589 8279 sys_catalog.cc:263] Verifying existing consensus state
I20251026 03:12:10.367941 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Bootstrap starting.
I20251026 03:12:10.370435 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Bootstrap replayed 1/1 log segments. Stats: ops{read=37 overwritten=0 applied=37 ignored=0} inserts{seen=5 ignored=0} mutations{seen=32 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:12:10.370645 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Bootstrap complete.
I20251026 03:12:10.370824 8279 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:12:10.370877 8279 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46176127b3ae434bbeba5dfb6dd21469, State: Initialized, Role: FOLLOWER
I20251026 03:12:10.370918 8279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:12:10.370949 8279 raft_consensus.cc:399] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:12:10.370968 8279 raft_consensus.cc:493] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:12:10.370991 8279 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:12:10.371433 8279 raft_consensus.cc:515] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:12:10.371479 8279 leader_election.cc:304] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46176127b3ae434bbeba5dfb6dd21469; no voters:
I20251026 03:12:10.371574 8279 leader_election.cc:290] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251026 03:12:10.371613 8282 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 03:12:10.371707 8282 raft_consensus.cc:697] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 2 LEADER]: Becoming Leader. State: Replica: 46176127b3ae434bbeba5dfb6dd21469, State: Running, Role: LEADER
I20251026 03:12:10.371706 8279 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:12:10.371778 8282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER }
I20251026 03:12:10.371942 8284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46176127b3ae434bbeba5dfb6dd21469. Latest consensus state: current_term: 2 leader_uuid: "46176127b3ae434bbeba5dfb6dd21469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER } }
I20251026 03:12:10.371940 8283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46176127b3ae434bbeba5dfb6dd21469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46176127b3ae434bbeba5dfb6dd21469" member_type: VOTER } }
I20251026 03:12:10.372020 8284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: This master's current role is: LEADER
I20251026 03:12:10.372028 8283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [sys.catalog]: This master's current role is: LEADER
I20251026 03:12:10.372169 8286 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:12:10.372414 8286 catalog_manager.cc:679] Loaded metadata for table test-workload [id=33b6cc56ebc2446da50a1ee0e341dec9]
I20251026 03:12:10.372534 8286 tablet_loader.cc:96] loaded metadata for tablet 64628560ddff4d0b94db2b5502de12e5 (table test-workload [id=33b6cc56ebc2446da50a1ee0e341dec9])
I20251026 03:12:10.372574 8286 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:12:10.372642 8286 catalog_manager.cc:1269] Loaded cluster ID: 8d4f5ea956f944a4be2d0daa0829611b
I20251026 03:12:10.372668 8286 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:12:10.372937 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:12:10.373400 8286 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:12:10.373494 8286 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469: Loaded TSK: 0
I20251026 03:12:10.373562 8286 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:12:10.374146 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:10.375205 8299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.375273 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:10.375171 8298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:10.375276 8301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.375463 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:10.375494 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:10.375514 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448330375514 us; error 0 us; skew 500 ppm
I20251026 03:12:10.375990 31233 webserver.cc:492] Webserver started at http://127.30.128.65:46013/ using document root <none> and password file <none>
I20251026 03:12:10.376062 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:10.376098 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:12:10.376539 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.376914 8306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.377028 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.377064 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "2a26a9c2ccd44b42a5d55fe3c9607040"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:12:10.377108 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:10.387959 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:10.388130 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:10.388257 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:10.388501 8313 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20251026 03:12:10.388970 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:12:10.389003 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:12:10.389032 31233 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251026 03:12:10.389408 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:12:10.389433 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.389441 8313 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap starting.
I20251026 03:12:10.392099 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:41819
I20251026 03:12:10.392477 8371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:41819 every 8 connection(s)
I20251026 03:12:10.392558 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:10.393589 8377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:10.393620 8376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.394264 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:10.394371 8379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.394554 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:10.394590 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:10.394611 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448330394612 us; error 0 us; skew 500 ppm
I20251026 03:12:10.395197 31233 webserver.cc:492] Webserver started at http://127.30.128.66:46449/ using document root <none> and password file <none>
I20251026 03:12:10.395278 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:10.395308 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:12:10.395913 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.396373 8385 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.396488 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.396539 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "5340c78feac8406882e860a2bed2ec2a"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:12:10.396580 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:10.399430 8372 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45221
I20251026 03:12:10.399484 8372 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:10.399578 8372 heartbeater.cc:507] Master 127.30.128.126:45221 requested a full tablet report, sending...
I20251026 03:12:10.399811 8243 ts_manager.cc:194] Registered new tserver with Master: 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:41819)
I20251026 03:12:10.400275 8243 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59210
I20251026 03:12:10.407905 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:10.408100 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:10.408242 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:10.408923 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:12:10.408977 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:12:10.409482 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:12:10.409525 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251026 03:12:10.409536 8392 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap starting.
I20251026 03:12:10.413064 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:39159
I20251026 03:12:10.413167 8450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:39159 every 8 connection(s)
I20251026 03:12:10.413378 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:10.414743 8457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:10.414765 8455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.415231 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:10.415383 8454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:10.415553 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:10.415622 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:10.415695 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448330415695 us; error 0 us; skew 500 ppm
I20251026 03:12:10.416388 31233 webserver.cc:492] Webserver started at http://127.30.128.67:34729/ using document root <none> and password file <none>
I20251026 03:12:10.416499 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:10.416568 31233 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 03:12:10.416656 8451 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45221
I20251026 03:12:10.416711 8451 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:10.416798 8451 heartbeater.cc:507] Master 127.30.128.126:45221 requested a full tablet report, sending...
I20251026 03:12:10.416998 8243 ts_manager.cc:194] Registered new tserver with Master: 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:39159)
I20251026 03:12:10.417390 8243 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59224
I20251026 03:12:10.418061 31233 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20251026 03:12:10.418690 8462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.418813 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251026 03:12:10.418850 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "360eb6050b7244d58989794aea84cc01"
format_stamp: "Formatted at 2025-10-26 03:11:56 on dist-test-slave-51kf"
I20251026 03:12:10.418896 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestFrequentElections.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:10.429499 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:10.429688 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:10.429894 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:10.430725 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251026 03:12:10.430771 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251026 03:12:10.431234 31233 ts_tablet_manager.cc:616] Registered 1 tablets
I20251026 03:12:10.431275 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:10.431349 8469 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap starting.
I20251026 03:12:10.434710 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:43339
I20251026 03:12:10.438115 8527 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:43339 every 8 connection(s)
I20251026 03:12:10.441547 8528 heartbeater.cc:344] Connected to a master server at 127.30.128.126:45221
I20251026 03:12:10.441605 8528 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:10.441694 8528 heartbeater.cc:507] Master 127.30.128.126:45221 requested a full tablet report, sending...
I20251026 03:12:10.441998 8243 ts_manager.cc:194] Registered new tserver with Master: 360eb6050b7244d58989794aea84cc01 (127.30.128.67:43339)
I20251026 03:12:10.442405 8243 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59236
I20251026 03:12:10.443054 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008197982s
I20251026 03:12:11.146807 8469 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap replayed 1/4 log segments. Stats: ops{read=4951 overwritten=0 applied=4949 ignored=0} inserts{seen=247400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:12:11.310617 8392 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap replayed 1/4 log segments. Stats: ops{read=4968 overwritten=0 applied=4966 ignored=0} inserts{seen=248250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:12:11.400691 8372 heartbeater.cc:499] Master 127.30.128.126:45221 was elected leader, sending a full tablet report...
I20251026 03:12:11.413214 8313 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap replayed 1/4 log segments. Stats: ops{read=4633 overwritten=0 applied=4630 ignored=0} inserts{seen=231450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:12:11.417827 8451 heartbeater.cc:499] Master 127.30.128.126:45221 was elected leader, sending a full tablet report...
I20251026 03:12:11.442898 8528 heartbeater.cc:499] Master 127.30.128.126:45221 was elected leader, sending a full tablet report...
I20251026 03:12:11.771409 8469 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap replayed 2/4 log segments. Stats: ops{read=9892 overwritten=0 applied=9890 ignored=0} inserts{seen=494450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:12:12.386333 8469 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap replayed 3/4 log segments. Stats: ops{read=14846 overwritten=0 applied=14843 ignored=0} inserts{seen=742100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:12:12.424256 8392 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap replayed 2/4 log segments. Stats: ops{read=9931 overwritten=0 applied=9929 ignored=0} inserts{seen=496400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20251026 03:12:12.459611 8313 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap replayed 2/4 log segments. Stats: ops{read=9263 overwritten=0 applied=9260 ignored=0} inserts{seen=462950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20251026 03:12:12.520689 8469 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap replayed 4/4 log segments. Stats: ops{read=15940 overwritten=3 applied=15936 ignored=0} inserts{seen=795251 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 03:12:12.521107 8469 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Bootstrap complete.
I20251026 03:12:12.526326 8469 ts_tablet_manager.cc:1403] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Time spent bootstrapping tablet: real 2.095s user 1.773s sys 0.320s
I20251026 03:12:12.526479 8469 raft_consensus.cc:359] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.526568 8469 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360eb6050b7244d58989794aea84cc01, State: Initialized, Role: FOLLOWER
I20251026 03:12:12.526620 8469 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15936, Last appended: 63.15937, Last appended by leader: 15937, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.526788 8469 ts_tablet_manager.cc:1434] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:12:12.552973 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:12.553026 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 63 FOLLOWER]: Advancing to term 64
I20251026 03:12:12.553606 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.553794 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 64 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:12.554451 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 64 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:12.554504 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 64 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:12.554528 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:12.554602 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20251026 03:12:12.585949 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:12.586006 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 64 FOLLOWER]: Advancing to term 65
I20251026 03:12:12.586491 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.586596 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 65 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:12.587067 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 65 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:12.587126 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 65 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:12.587152 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 65 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:12.587225 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 65 FOLLOWER]: Leader election lost for term 65. Reason: could not achieve majority
I20251026 03:12:12.630713 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 65 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:12.630787 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 65 FOLLOWER]: Advancing to term 66
I20251026 03:12:12.631492 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.631626 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 66 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:12.632208 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 66 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:12.632266 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 66 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:12.632293 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:12.632368 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20251026 03:12:12.703935 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 66 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:12.704020 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 66 FOLLOWER]: Advancing to term 67
I20251026 03:12:12.704721 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 67 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.704869 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 67 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:12.705477 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 67 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:12.705535 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 67 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:12.705564 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 67 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:12.705636 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 67 FOLLOWER]: Leader election lost for term 67. Reason: could not achieve majority
I20251026 03:12:12.742033 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 67 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:12.742095 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 67 FOLLOWER]: Advancing to term 68
I20251026 03:12:12.742697 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 68 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.742827 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 68 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:12.743345 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 68 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
W20251026 03:12:12.743402 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 68 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
I20251026 03:12:12.743425 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 68 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:12.743503 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 68 FOLLOWER]: Leader election lost for term 68. Reason: could not achieve majority
I20251026 03:12:12.917541 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 68 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:12.917634 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 68 FOLLOWER]: Advancing to term 69
I20251026 03:12:12.918466 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 69 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:12.918622 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 69 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:12.919384 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 69 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:12.919463 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 69 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:12.919495 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 69 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:12.919585 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 69 FOLLOWER]: Leader election lost for term 69. Reason: could not achieve majority
I20251026 03:12:13.038444 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 69 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:13.038534 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 69 FOLLOWER]: Advancing to term 70
I20251026 03:12:13.039330 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 70 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.039490 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 70 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:13.040040 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 70 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:13.040167 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 70 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:13.040194 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 70 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:13.040290 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 70 FOLLOWER]: Leader election lost for term 70. Reason: could not achieve majority
I20251026 03:12:13.129241 8313 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap replayed 3/4 log segments. Stats: ops{read=13896 overwritten=0 applied=13892 ignored=0} inserts{seen=694550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20251026 03:12:13.143577 8392 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap replayed 3/4 log segments. Stats: ops{read=14889 overwritten=0 applied=14889 ignored=0} inserts{seen=744400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:12:13.272754 8539 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 70 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:13.272843 8539 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 70 FOLLOWER]: Advancing to term 71
I20251026 03:12:13.273672 8539 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 71 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.273881 8539 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 71 election: Requested vote from peers 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
I20251026 03:12:13.273983 8392 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap replayed 4/4 log segments. Stats: ops{read=15936 overwritten=0 applied=15936 ignored=0} inserts{seen=795251 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:12:13.274399 8392 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Bootstrap complete.
W20251026 03:12:13.274565 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 71 election: RPC error from VoteRequest() call to peer 5340c78feac8406882e860a2bed2ec2a (127.30.128.66:32837): Network error: Client connection negotiation failed: client connection to 127.30.128.66:32837: connect: Connection refused (error 111)
W20251026 03:12:13.274618 8466 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 71 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:13.274637 8466 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [CANDIDATE]: Term 71 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 360eb6050b7244d58989794aea84cc01; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 5340c78feac8406882e860a2bed2ec2a
I20251026 03:12:13.274710 8539 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 71 FOLLOWER]: Leader election lost for term 71. Reason: could not achieve majority
I20251026 03:12:13.279565 8392 ts_tablet_manager.cc:1403] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Time spent bootstrapping tablet: real 2.870s user 2.423s sys 0.443s
I20251026 03:12:13.279681 8392 raft_consensus.cc:359] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.279737 8392 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5340c78feac8406882e860a2bed2ec2a, State: Initialized, Role: FOLLOWER
I20251026 03:12:13.279773 8392 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15936, Last appended: 60.15936, Last appended by leader: 15936, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.279914 8392 ts_tablet_manager.cc:1434] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20251026 03:12:13.304147 8545 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:13.304201 8545 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 63 FOLLOWER]: Advancing to term 64
I20251026 03:12:13.304780 8545 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 64 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.305109 8545 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 64 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:13.305640 8387 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 64 election: RPC error from VoteRequest() call to peer 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131): Network error: Client connection negotiation failed: client connection to 127.30.128.67:37131: connect: Connection refused (error 111)
W20251026 03:12:13.305722 8389 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 64 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:13.305799 8389 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 64 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:13.305886 8545 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 64 FOLLOWER]: Leader election lost for term 64. Reason: could not achieve majority
I20251026 03:12:13.347302 8545 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 64 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:13.347357 8545 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 64 FOLLOWER]: Advancing to term 65
I20251026 03:12:13.347872 8545 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 65 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.347983 8545 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 65 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:13.348532 8389 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 65 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
W20251026 03:12:13.348610 8387 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 65 election: RPC error from VoteRequest() call to peer 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131): Network error: Client connection negotiation failed: client connection to 127.30.128.67:37131: connect: Connection refused (error 111)
I20251026 03:12:13.348647 8387 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 65 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:13.348744 8545 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 65 FOLLOWER]: Leader election lost for term 65. Reason: could not achieve majority
I20251026 03:12:13.392670 8313 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap replayed 4/4 log segments. Stats: ops{read=15938 overwritten=2 applied=15936 ignored=0} inserts{seen=795251 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 03:12:13.393049 8313 tablet_bootstrap.cc:492] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Bootstrap complete.
I20251026 03:12:13.398114 8313 ts_tablet_manager.cc:1403] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Time spent bootstrapping tablet: real 3.009s user 2.546s sys 0.451s
I20251026 03:12:13.398283 8313 raft_consensus.cc:359] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 61 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.398345 8313 raft_consensus.cc:740] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 61 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a26a9c2ccd44b42a5d55fe3c9607040, State: Initialized, Role: FOLLOWER
I20251026 03:12:13.398402 8313 consensus_queue.cc:260] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15936, Last appended: 60.15936, Last appended by leader: 15936, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.398576 8313 ts_tablet_manager.cc:1434] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:12:13.398738 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:12:13.400832 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:13.400960 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:12:13.401000 31233 raft_consensus.cc:2243] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 61 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:13.401048 31233 raft_consensus.cc:2272] T 64628560ddff4d0b94db2b5502de12e5 P 2a26a9c2ccd44b42a5d55fe3c9607040 [term 61 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:13.413232 8545 raft_consensus.cc:493] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 65 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:13.413281 8545 raft_consensus.cc:3060] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 65 FOLLOWER]: Advancing to term 66
I20251026 03:12:13.413841 8545 raft_consensus.cc:515] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 66 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360eb6050b7244d58989794aea84cc01" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37131 } } peers { permanent_uuid: "5340c78feac8406882e860a2bed2ec2a" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 32837 } } peers { permanent_uuid: "2a26a9c2ccd44b42a5d55fe3c9607040" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 46839 } }
I20251026 03:12:13.413957 8545 leader_election.cc:290] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 66 election: Requested vote from peers 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131), 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839)
W20251026 03:12:13.414417 8387 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 66 election: RPC error from VoteRequest() call to peer 360eb6050b7244d58989794aea84cc01 (127.30.128.67:37131): Network error: Client connection negotiation failed: client connection to 127.30.128.67:37131: connect: Connection refused (error 111)
W20251026 03:12:13.414463 8389 leader_election.cc:336] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 66 election: RPC error from VoteRequest() call to peer 2a26a9c2ccd44b42a5d55fe3c9607040 (127.30.128.65:46839): Network error: Client connection negotiation failed: client connection to 127.30.128.65:46839: connect: Connection refused (error 111)
I20251026 03:12:13.414497 8389 leader_election.cc:304] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [CANDIDATE]: Term 66 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5340c78feac8406882e860a2bed2ec2a; no voters: 2a26a9c2ccd44b42a5d55fe3c9607040, 360eb6050b7244d58989794aea84cc01
I20251026 03:12:13.414548 8545 raft_consensus.cc:2749] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 66 FOLLOWER]: Leader election lost for term 66. Reason: could not achieve majority
I20251026 03:12:13.418205 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:12:13.418953 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:12:13.420864 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:13.420992 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:12:13.421061 31233 raft_consensus.cc:2243] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 66 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:13.421100 31233 raft_consensus.cc:2272] T 64628560ddff4d0b94db2b5502de12e5 P 5340c78feac8406882e860a2bed2ec2a [term 66 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:13.434406 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:12:13.435073 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:12:13.437803 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:13.437896 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:12:13.437939 31233 raft_consensus.cc:2243] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 71 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:13.437980 31233 pending_rounds.cc:70] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Trying to abort 1 pending ops.
I20251026 03:12:13.438004 31233 pending_rounds.cc:77] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01: Aborting op as it isn't in flight: id { term: 63 index: 15937 } timestamp: 7214892360813027328 op_type: NO_OP noop_request { }
I20251026 03:12:13.438045 31233 raft_consensus.cc:2889] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 71 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251026 03:12:13.438072 31233 raft_consensus.cc:2272] T 64628560ddff4d0b94db2b5502de12e5 P 360eb6050b7244d58989794aea84cc01 [term 71 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:13.460167 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:12:13.460826 31233 master.cc:561] Master@127.30.128.126:45221 shutting down...
I20251026 03:12:13.462489 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 2 LEADER]: Raft consensus shutting down.
I20251026 03:12:13.462533 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 46176127b3ae434bbeba5dfb6dd21469 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:13.462548 31233 tablet_replica.cc:333] stopping tablet replica
I20251026 03:12:13.473639 31233 master.cc:583] Master@127.30.128.126:45221 shutdown complete.
[ OK ] TxnParticipantElectionStormITest.TestFrequentElections (16766 ms)
[ RUN ] TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm
I20251026 03:12:13.492841 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:32851
I20251026 03:12:13.492978 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:13.493995 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:13.494060 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:13.494060 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.494112 31233 server_base.cc:1047] running on GCE node
I20251026 03:12:13.494323 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:13.494355 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:13.494377 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448333494377 us; error 0 us; skew 500 ppm
I20251026 03:12:13.494921 31233 webserver.cc:492] Webserver started at http://127.30.128.126:41311/ using document root <none> and password file <none>
I20251026 03:12:13.494998 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:13.495047 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:13.495092 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:13.495314 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "9fa7c9b520aa416896fa8615540ab310"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.496074 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:12:13.496430 8558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.496544 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.496582 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "9fa7c9b520aa416896fa8615540ab310"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.496624 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:13.510064 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:13.510214 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:13.512306 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:32851
I20251026 03:12:13.513298 8610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:32851 every 8 connection(s)
I20251026 03:12:13.517894 8611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:13.519865 8611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310: Bootstrap starting.
I20251026 03:12:13.520113 8611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:13.520498 8611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310: No bootstrap required, opened a new log
I20251026 03:12:13.520597 8611 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa7c9b520aa416896fa8615540ab310" member_type: VOTER }
I20251026 03:12:13.520638 8611 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:13.520650 8611 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa7c9b520aa416896fa8615540ab310, State: Initialized, Role: FOLLOWER
I20251026 03:12:13.520678 8611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa7c9b520aa416896fa8615540ab310" member_type: VOTER }
I20251026 03:12:13.520713 8611 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:12:13.520735 8611 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:12:13.520757 8611 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:13.521165 8611 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa7c9b520aa416896fa8615540ab310" member_type: VOTER }
I20251026 03:12:13.521211 8611 leader_election.cc:304] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa7c9b520aa416896fa8615540ab310; no voters:
I20251026 03:12:13.521317 8611 leader_election.cc:290] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:12:13.521390 8614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:12:13.521435 8611 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:12:13.521515 8614 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa7c9b520aa416896fa8615540ab310, State: Running, Role: LEADER
I20251026 03:12:13.521565 8614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa7c9b520aa416896fa8615540ab310" member_type: VOTER }
I20251026 03:12:13.521759 8615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa7c9b520aa416896fa8615540ab310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa7c9b520aa416896fa8615540ab310" member_type: VOTER } }
I20251026 03:12:13.521780 8616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa7c9b520aa416896fa8615540ab310. Latest consensus state: current_term: 1 leader_uuid: "9fa7c9b520aa416896fa8615540ab310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa7c9b520aa416896fa8615540ab310" member_type: VOTER } }
I20251026 03:12:13.521852 8615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [sys.catalog]: This master's current role is: LEADER
I20251026 03:12:13.521855 8616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [sys.catalog]: This master's current role is: LEADER
I20251026 03:12:13.522116 8621 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:12:13.522244 8621 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:12:13.522733 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:12:13.522759 8621 catalog_manager.cc:1357] Generated new cluster ID: 0ca0ee427eb5410ba5f545fb107ee5c2
I20251026 03:12:13.522799 8621 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:12:13.527730 8621 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:12:13.528097 8621 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:12:13.531368 8621 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310: Generated new TSK 0
I20251026 03:12:13.531433 8621 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:12:13.538242 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:13.539405 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:13.539443 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.539548 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:13.539542 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.539726 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:13.539759 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:13.539775 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448333539775 us; error 0 us; skew 500 ppm
I20251026 03:12:13.540241 31233 webserver.cc:492] Webserver started at http://127.30.128.65:44397/ using document root <none> and password file <none>
I20251026 03:12:13.540318 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:13.540347 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:13.540378 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:13.540562 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "417cdd6c915c45188264c5da00328e12"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.541339 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:13.541887 8640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.542037 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.542083 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "417cdd6c915c45188264c5da00328e12"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.542122 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:13.553606 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:13.553763 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:13.553923 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:13.554083 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:12:13.554105 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.554132 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:12:13.554153 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.556564 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:43855
I20251026 03:12:13.556588 8703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:43855 every 8 connection(s)
I20251026 03:12:13.556852 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:13.557858 8708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:13.557889 8709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.559221 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:13.558018 8711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.559455 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:13.559506 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:13.559523 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448333559523 us; error 0 us; skew 500 ppm
I20251026 03:12:13.560065 31233 webserver.cc:492] Webserver started at http://127.30.128.66:39149/ using document root <none> and password file <none>
I20251026 03:12:13.560145 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:13.560185 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:13.560225 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:13.560293 8704 heartbeater.cc:344] Connected to a master server at 127.30.128.126:32851
I20251026 03:12:13.560348 8704 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:13.560428 8704 heartbeater.cc:507] Master 127.30.128.126:32851 requested a full tablet report, sending...
I20251026 03:12:13.560462 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.560660 8575 ts_manager.cc:194] Registered new tserver with Master: 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855)
I20251026 03:12:13.561129 8575 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53930
I20251026 03:12:13.561344 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:13.561807 8716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.561923 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.561960 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.562000 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:13.570477 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:13.570621 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:13.570750 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:13.570922 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:12:13.570948 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.570976 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:12:13.571004 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.573552 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:41919
I20251026 03:12:13.573611 8779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:41919 every 8 connection(s)
I20251026 03:12:13.573905 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:13.574894 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.575136 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:13.576007 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:13.576438 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:13.576711 8780 heartbeater.cc:344] Connected to a master server at 127.30.128.126:32851
I20251026 03:12:13.576789 8780 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:13.576898 8780 heartbeater.cc:507] Master 127.30.128.126:32851 requested a full tablet report, sending...
I20251026 03:12:13.577129 8575 ts_manager.cc:194] Registered new tserver with Master: c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:13.577137 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:13.577236 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:13.577297 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448333577297 us; error 0 us; skew 500 ppm
I20251026 03:12:13.577545 8575 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53944
I20251026 03:12:13.577909 31233 webserver.cc:492] Webserver started at http://127.30.128.67:35257/ using document root <none> and password file <none>
I20251026 03:12:13.577983 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:13.578022 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:13.578061 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:13.578290 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "60247008d7194489970aea59d5aa5c7a"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.579533 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 03:12:13.580015 8791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.580149 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.580193 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "60247008d7194489970aea59d5aa5c7a"
format_stamp: "Formatted at 2025-10-26 03:12:13 on dist-test-slave-51kf"
I20251026 03:12:13.580235 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:13.589730 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:13.589902 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:13.590003 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:13.590116 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:12:13.590137 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.590157 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:12:13.590173 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:13.592809 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:37853
I20251026 03:12:13.592859 8854 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:37853 every 8 connection(s)
I20251026 03:12:13.595075 8855 heartbeater.cc:344] Connected to a master server at 127.30.128.126:32851
I20251026 03:12:13.595118 8855 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:13.595189 8855 heartbeater.cc:507] Master 127.30.128.126:32851 requested a full tablet report, sending...
I20251026 03:12:13.595373 8575 ts_manager.cc:194] Registered new tserver with Master: 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:13.595628 8575 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53952
I20251026 03:12:13.596215 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003254632s
I20251026 03:12:13.596455 31233 test_util.cc:276] Using random seed: 641128231
I20251026 03:12:13.600170 8575 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53956:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:12:13.600274 8575 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:12:13.604085 8668 tablet_service.cc:1505] Processing CreateTablet for tablet 3345c67fff444b66ab9a03779e1022cb (DEFAULT_TABLE table=test-workload [id=1ff166329c9746d7ad1d1b9a37d8d859]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:12:13.604193 8668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3345c67fff444b66ab9a03779e1022cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:13.604409 8744 tablet_service.cc:1505] Processing CreateTablet for tablet 3345c67fff444b66ab9a03779e1022cb (DEFAULT_TABLE table=test-workload [id=1ff166329c9746d7ad1d1b9a37d8d859]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:12:13.604488 8744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3345c67fff444b66ab9a03779e1022cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:13.605459 8870 tablet_bootstrap.cc:492] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12: Bootstrap starting.
I20251026 03:12:13.605664 8871 tablet_bootstrap.cc:492] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: Bootstrap starting.
I20251026 03:12:13.605878 8870 tablet_bootstrap.cc:654] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:13.606071 8819 tablet_service.cc:1505] Processing CreateTablet for tablet 3345c67fff444b66ab9a03779e1022cb (DEFAULT_TABLE table=test-workload [id=1ff166329c9746d7ad1d1b9a37d8d859]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:12:13.606083 8871 tablet_bootstrap.cc:654] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:13.606170 8819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3345c67fff444b66ab9a03779e1022cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:13.606577 8871 tablet_bootstrap.cc:492] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: No bootstrap required, opened a new log
I20251026 03:12:13.606626 8871 ts_tablet_manager.cc:1403] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:13.606765 8871 raft_consensus.cc:359] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.606822 8871 raft_consensus.cc:385] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:13.606849 8871 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7769e7ca93b413d92d7ec142d7b26b7, State: Initialized, Role: FOLLOWER
I20251026 03:12:13.606897 8871 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.607049 8871 ts_tablet_manager.cc:1434] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:12:13.607081 8780 heartbeater.cc:499] Master 127.30.128.126:32851 was elected leader, sending a full tablet report...
I20251026 03:12:13.607225 8870 tablet_bootstrap.cc:492] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12: No bootstrap required, opened a new log
I20251026 03:12:13.607270 8870 ts_tablet_manager.cc:1403] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251026 03:12:13.607323 8875 tablet_bootstrap.cc:492] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a: Bootstrap starting.
I20251026 03:12:13.607395 8870 raft_consensus.cc:359] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.607530 8870 raft_consensus.cc:385] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:13.607549 8870 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417cdd6c915c45188264c5da00328e12, State: Initialized, Role: FOLLOWER
I20251026 03:12:13.607589 8870 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.607714 8870 ts_tablet_manager.cc:1434] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251026 03:12:13.607792 8704 heartbeater.cc:499] Master 127.30.128.126:32851 was elected leader, sending a full tablet report...
I20251026 03:12:13.607805 8875 tablet_bootstrap.cc:654] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:13.608346 8875 tablet_bootstrap.cc:492] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a: No bootstrap required, opened a new log
I20251026 03:12:13.608398 8875 ts_tablet_manager.cc:1403] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:13.608520 8875 raft_consensus.cc:359] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.608577 8875 raft_consensus.cc:385] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:13.608595 8875 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60247008d7194489970aea59d5aa5c7a, State: Initialized, Role: FOLLOWER
I20251026 03:12:13.608646 8875 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.608784 8875 ts_tablet_manager.cc:1434] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:12:13.608817 8855 heartbeater.cc:499] Master 127.30.128.126:32851 was elected leader, sending a full tablet report...
I20251026 03:12:13.609736 8876 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:12:13.609812 8876 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.609927 8876 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:13.612210 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a" is_pre_election: true
I20251026 03:12:13.612270 8829 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417cdd6c915c45188264c5da00328e12 in term 0.
I20251026 03:12:13.612326 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" is_pre_election: true
I20251026 03:12:13.612392 8644 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a; no voters:
I20251026 03:12:13.612393 8754 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 417cdd6c915c45188264c5da00328e12 in term 0.
I20251026 03:12:13.612484 8876 raft_consensus.cc:2804] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:12:13.612524 8876 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:13.612552 8876 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:13.612993 8876 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.613082 8876 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 1 election: Requested vote from peers 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:13.613198 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:13.613255 8829 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:13.613212 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:13.613284 8754 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:13.613744 8829 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417cdd6c915c45188264c5da00328e12 in term 1.
I20251026 03:12:13.613852 8754 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417cdd6c915c45188264c5da00328e12 in term 1.
I20251026 03:12:13.613925 8644 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a; no voters:
I20251026 03:12:13.614001 8876 raft_consensus.cc:2804] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:12:13.614105 8876 raft_consensus.cc:697] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 1 LEADER]: Becoming Leader. State: Replica: 417cdd6c915c45188264c5da00328e12, State: Running, Role: LEADER
I20251026 03:12:13.614151 8876 consensus_queue.cc:237] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:13.614514 8574 catalog_manager.cc:5649] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 417cdd6c915c45188264c5da00328e12 (127.30.128.65). New cstate: current_term: 1 leader_uuid: "417cdd6c915c45188264c5da00328e12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:13.628438 8754 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 1 FOLLOWER]: Refusing update from remote peer 417cdd6c915c45188264c5da00328e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:12:13.628448 8829 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 1 FOLLOWER]: Refusing update from remote peer 417cdd6c915c45188264c5da00328e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:12:13.628629 8876 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:12:13.628705 8880 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:12:23.651894 8928 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 417cdd6c915c45188264c5da00328e12)
I20251026 03:12:23.651957 8928 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 1 FOLLOWER]: Advancing to term 2
I20251026 03:12:23.652530 8928 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.652670 8928 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 2 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:23.655048 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 16065 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:23.655050 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 2 candidate_status { last_received { term: 1 index: 16065 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.655303 8717 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7769e7ca93b413d92d7ec142d7b26b7; no voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a
I20251026 03:12:23.655385 8928 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 03:12:23.656318 8754 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 2 FOLLOWER]: Rejecting Update request from peer 417cdd6c915c45188264c5da00328e12 for earlier term 1. Current term is 2. Ops: []
I20251026 03:12:23.656518 8880 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: INVALID_TERM, Last received: 1.16065, Next index: 16066, Last known committed idx: 16065, Time since last communication: 0.000s
I20251026 03:12:23.677793 8880 raft_consensus.cc:3055] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 03:12:23.677840 8880 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 417cdd6c915c45188264c5da00328e12, State: Running, Role: LEADER
I20251026 03:12:23.677884 8880 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16065, Committed index: 16065, Last appended: 1.16065, Last appended by leader: 16065, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.677963 8880 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 1 FOLLOWER]: Advancing to term 2
W20251026 03:12:23.687654 8668 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39776: Illegal state: replica 417cdd6c915c45188264c5da00328e12 is not leader of this config: current role FOLLOWER
I20251026 03:12:23.688808 8928 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:23.688848 8928 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:12:23.689311 8928 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.689414 8928 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 3 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:23.689553 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 16065 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:23.689563 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 3 candidate_status { last_received { term: 1 index: 16065 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.689667 8678 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 2 FOLLOWER]: Advancing to term 3
I20251026 03:12:23.690475 8678 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7769e7ca93b413d92d7ec142d7b26b7 in term 3.
I20251026 03:12:23.690604 8717 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7; no voters: 60247008d7194489970aea59d5aa5c7a
I20251026 03:12:23.690691 8928 raft_consensus.cc:2804] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 03:12:23.690805 8928 raft_consensus.cc:697] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 3 LEADER]: Becoming Leader. State: Replica: c7769e7ca93b413d92d7ec142d7b26b7, State: Running, Role: LEADER
I20251026 03:12:23.690860 8928 consensus_queue.cc:237] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16065, Committed index: 16065, Last appended: 1.16065, Last appended by leader: 16065, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.691262 8575 catalog_manager.cc:5649] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 reported cstate change: term changed from 1 to 3, leader changed from 417cdd6c915c45188264c5da00328e12 (127.30.128.65) to c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66). New cstate: current_term: 3 leader_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } health_report { overall_health: HEALTHY } } }
W20251026 03:12:23.692075 8819 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36754: Illegal state: replica 60247008d7194489970aea59d5aa5c7a is not leader of this config: current role FOLLOWER
I20251026 03:12:23.697420 8829 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 1 FOLLOWER]: Advancing to term 3
I20251026 03:12:23.697520 8678 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 3 FOLLOWER]: Refusing update from remote peer c7769e7ca93b413d92d7ec142d7b26b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 16065. Preceding OpId from leader: term: 3 index: 16067. (index mismatch)
I20251026 03:12:23.697674 8934 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16066, Last known committed idx: 16065, Time since last communication: 0.000s
I20251026 03:12:23.698148 8829 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 3 FOLLOWER]: Refusing update from remote peer c7769e7ca93b413d92d7ec142d7b26b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 16065. Preceding OpId from leader: term: 3 index: 16067. (index mismatch)
I20251026 03:12:23.698292 8934 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16066, Last known committed idx: 16065, Time since last communication: 0.000s
I20251026 03:12:23.737305 8880 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c7769e7ca93b413d92d7ec142d7b26b7)
I20251026 03:12:23.737351 8880 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:12:23.737607 8940 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 3 FOLLOWER]: Starting leader election (detected failure of leader c7769e7ca93b413d92d7ec142d7b26b7)
I20251026 03:12:23.737653 8940 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 3 FOLLOWER]: Advancing to term 4
I20251026 03:12:23.737881 8880 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.737977 8880 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 4 election: Requested vote from peers 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:23.738087 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 4 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:23.738116 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 4 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:23.738268 8940 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.738363 8829 raft_consensus.cc:2393] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 417cdd6c915c45188264c5da00328e12 in current term 4: Already voted for candidate 60247008d7194489970aea59d5aa5c7a in this term.
I20251026 03:12:23.738412 8940 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 4 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:23.738799 8644 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 417cdd6c915c45188264c5da00328e12; no voters: 60247008d7194489970aea59d5aa5c7a, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:23.738884 8880 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 03:12:23.738945 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 4 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 3. Current term is 4. Ops: []
I20251026 03:12:23.739086 8939 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.740290 8678 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 4 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 3. Current term is 4. Ops: []
I20251026 03:12:23.740473 8928 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.741179 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.741255 8678 raft_consensus.cc:2393] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60247008d7194489970aea59d5aa5c7a in current term 4: Already voted for candidate 417cdd6c915c45188264c5da00328e12 in this term.
I20251026 03:12:23.741358 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:23.741479 8795 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60247008d7194489970aea59d5aa5c7a; no voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:23.741559 8940 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251026 03:12:23.770416 8880 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:23.770462 8880 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 4 FOLLOWER]: Advancing to term 5
I20251026 03:12:23.770862 8880 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.770946 8880 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 5 election: Requested vote from peers 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:23.771104 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 5 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:23.771116 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 5 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:23.771162 8829 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 4 FOLLOWER]: Advancing to term 5
I20251026 03:12:23.771687 8829 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 417cdd6c915c45188264c5da00328e12 in term 5.
I20251026 03:12:23.771839 8644 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a; no voters: c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:23.771932 8880 raft_consensus.cc:2804] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 03:12:23.771986 8880 raft_consensus.cc:697] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 LEADER]: Becoming Leader. State: Replica: 417cdd6c915c45188264c5da00328e12, State: Running, Role: LEADER
I20251026 03:12:23.772032 8880 consensus_queue.cc:237] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16068, Committed index: 16068, Last appended: 3.16068, Last appended by leader: 16068, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.772398 8573 catalog_manager.cc:5649] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 reported cstate change: term changed from 3 to 5, leader changed from c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66) to 417cdd6c915c45188264c5da00328e12 (127.30.128.65). New cstate: current_term: 5 leader_uuid: "417cdd6c915c45188264c5da00328e12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:23.772528 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 5 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 3. Current term is 5. Ops: []
I20251026 03:12:23.772691 8934 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.774812 8678 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 LEADER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 3. Current term is 5. Ops: []
I20251026 03:12:23.774986 8928 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.775053 8928 consensus_queue.cc:1243] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer 417cdd6c915c45188264c5da00328e12 log is divergent from this leader: its last log entry 5.16069 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16068
I20251026 03:12:23.796386 8754 raft_consensus.cc:3055] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 03:12:23.796425 8754 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c7769e7ca93b413d92d7ec142d7b26b7, State: Running, Role: LEADER
I20251026 03:12:23.796463 8754 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16068, Committed index: 16068, Last appended: 3.16068, Last appended by leader: 16068, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.796542 8754 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 3 FOLLOWER]: Advancing to term 5
I20251026 03:12:23.797036 8754 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 5 FOLLOWER]: Refusing update from remote peer 417cdd6c915c45188264c5da00328e12: Log matching property violated. Preceding OpId in replica: term: 3 index: 16068. Preceding OpId from leader: term: 5 index: 16069. (index mismatch)
I20251026 03:12:23.797245 8880 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
W20251026 03:12:23.797823 8744 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35590: Illegal state: replica c7769e7ca93b413d92d7ec142d7b26b7 is not leader of this config: current role FOLLOWER
I20251026 03:12:23.801741 8829 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 5 FOLLOWER]: Refusing update from remote peer 417cdd6c915c45188264c5da00328e12: Log matching property violated. Preceding OpId in replica: term: 3 index: 16068. Preceding OpId from leader: term: 5 index: 16070. (index mismatch)
I20251026 03:12:23.801939 8880 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.802079 8940 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:23.802114 8940 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 5 FOLLOWER]: Advancing to term 6
I20251026 03:12:23.802587 8940 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.802686 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 6 FOLLOWER]: Rejecting Update request from peer 417cdd6c915c45188264c5da00328e12 for earlier term 5. Current term is 6. Ops: [5.16069-5.16070]
I20251026 03:12:23.802701 8940 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 6 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:23.802826 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 6 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.802919 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 6 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:23.802999 8921 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.803046 8795 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60247008d7194489970aea59d5aa5c7a; no voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:23.803120 8940 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251026 03:12:23.828164 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 6 FOLLOWER]: Rejecting Update request from peer 417cdd6c915c45188264c5da00328e12 for earlier term 5. Current term is 6. Ops: [5.16069-5.16070]
I20251026 03:12:23.828351 8921 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.852520 8940 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:23.852569 8940 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 6 FOLLOWER]: Advancing to term 7
I20251026 03:12:23.853025 8940 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.853138 8940 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 7 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:23.853262 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 7 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.853284 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 7 candidate_status { last_received { term: 3 index: 16068 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:23.853430 8795 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60247008d7194489970aea59d5aa5c7a; no voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:23.853541 8940 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251026 03:12:23.863332 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Rejecting Update request from peer 417cdd6c915c45188264c5da00328e12 for earlier term 5. Current term is 7. Ops: [5.16069-5.16070]
I20251026 03:12:23.863499 8904 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.865427 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Rejecting Update request from peer 417cdd6c915c45188264c5da00328e12 for earlier term 5. Current term is 7. Ops: [5.16069-5.16070]
I20251026 03:12:23.865636 8921 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.865932 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Rejecting Update request from peer 417cdd6c915c45188264c5da00328e12 for earlier term 5. Current term is 7. Ops: [5.16069-5.16071]
I20251026 03:12:23.866129 8904 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 3.16068, Next index: 16069, Last known committed idx: 16068, Time since last communication: 0.000s
I20251026 03:12:23.890355 8917 raft_consensus.cc:3055] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 LEADER]: Stepping down as leader of term 5
I20251026 03:12:23.890417 8917 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 417cdd6c915c45188264c5da00328e12, State: Running, Role: LEADER
I20251026 03:12:23.890460 8917 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16071, Committed index: 16071, Last appended: 5.16071, Last appended by leader: 16071, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.890542 8917 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 5 FOLLOWER]: Advancing to term 6
I20251026 03:12:23.896828 8928 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 417cdd6c915c45188264c5da00328e12)
I20251026 03:12:23.896878 8928 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 5 FOLLOWER]: Advancing to term 6
I20251026 03:12:23.897362 8928 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.897480 8928 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 6 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:23.897594 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 6 candidate_status { last_received { term: 5 index: 16071 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.897616 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 6 candidate_status { last_received { term: 5 index: 16071 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:23.897686 8829 raft_consensus.cc:2368] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 6. Current term is 7.
I20251026 03:12:23.897858 8720 leader_election.cc:400] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 6 election: Vote denied by peer 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853) with higher term. Message: Invalid argument: T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 6. Current term is 7.
I20251026 03:12:23.897953 8720 leader_election.cc:403] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20251026 03:12:23.898010 8928 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 6 FOLLOWER]: Advancing to term 7
I20251026 03:12:23.898276 8678 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7769e7ca93b413d92d7ec142d7b26b7 in term 6.
I20251026 03:12:23.898469 8928 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853) with higher term. Message: Invalid argument: T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 6. Current term is 7.
I20251026 03:12:23.928848 8928 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:23.928891 8928 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 7 FOLLOWER]: Advancing to term 8
I20251026 03:12:23.929297 8928 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.929402 8928 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 8 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:23.929574 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 8 candidate_status { last_received { term: 5 index: 16071 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.929574 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 8 candidate_status { last_received { term: 5 index: 16071 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:23.929678 8678 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 6 FOLLOWER]: Advancing to term 8
I20251026 03:12:23.929683 8829 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 7 FOLLOWER]: Advancing to term 8
I20251026 03:12:23.930341 8829 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7769e7ca93b413d92d7ec142d7b26b7 in term 8.
I20251026 03:12:23.930341 8678 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7769e7ca93b413d92d7ec142d7b26b7 in term 8.
I20251026 03:12:23.930498 8717 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7; no voters:
I20251026 03:12:23.930585 8928 raft_consensus.cc:2804] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 8 FOLLOWER]: Leader election won for term 8
I20251026 03:12:23.930640 8928 raft_consensus.cc:697] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 8 LEADER]: Becoming Leader. State: Replica: c7769e7ca93b413d92d7ec142d7b26b7, State: Running, Role: LEADER
I20251026 03:12:23.930683 8928 consensus_queue.cc:237] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16071, Committed index: 16071, Last appended: 5.16071, Last appended by leader: 16071, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.931005 8574 catalog_manager.cc:5649] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 reported cstate change: term changed from 5 to 8, leader changed from 417cdd6c915c45188264c5da00328e12 (127.30.128.65) to c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66). New cstate: current_term: 8 leader_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } health_report { overall_health: HEALTHY } } }
I20251026 03:12:23.955221 8678 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 8 FOLLOWER]: Refusing update from remote peer c7769e7ca93b413d92d7ec142d7b26b7: Log matching property violated. Preceding OpId in replica: term: 5 index: 16071. Preceding OpId from leader: term: 8 index: 16072. (index mismatch)
I20251026 03:12:23.955389 8928 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16072, Last known committed idx: 16071, Time since last communication: 0.000s
I20251026 03:12:23.955442 8829 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 8 FOLLOWER]: Refusing update from remote peer c7769e7ca93b413d92d7ec142d7b26b7: Log matching property violated. Preceding OpId in replica: term: 3 index: 16068. Preceding OpId from leader: term: 8 index: 16072. (index mismatch)
I20251026 03:12:23.955577 8928 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16072, Last known committed idx: 16068, Time since last communication: 0.000s
W20251026 03:12:23.956030 8668 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39776: Illegal state: replica 417cdd6c915c45188264c5da00328e12 is not leader of this config: current role FOLLOWER
W20251026 03:12:23.960479 8819 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36754: Illegal state: replica 60247008d7194489970aea59d5aa5c7a is not leader of this config: current role FOLLOWER
W20251026 03:12:23.992799 8939 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 8 is higher.
I20251026 03:12:23.997025 8944 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 8 FOLLOWER]: Starting leader election (detected failure of leader c7769e7ca93b413d92d7ec142d7b26b7)
I20251026 03:12:23.997071 8944 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 8 FOLLOWER]: Advancing to term 9
I20251026 03:12:23.997578 8944 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:23.997695 8944 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 9 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:23.997843 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 9 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:23.997869 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 9 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:23.998018 8795 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60247008d7194489970aea59d5aa5c7a; no voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:23.998104 8944 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251026 03:12:24.000959 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 9 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 9. Ops: []
I20251026 03:12:24.001139 8934 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.029418 8944 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:24.029481 8944 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 9 FOLLOWER]: Advancing to term 10
I20251026 03:12:24.030043 8944 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.030164 8944 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 10 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:24.030325 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 10 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:24.030325 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 10 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:24.030645 8795 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60247008d7194489970aea59d5aa5c7a; no voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:24.030721 8944 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251026 03:12:24.036298 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 10. Ops: []
I20251026 03:12:24.036486 8928 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.068423 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 10. Ops: []
I20251026 03:12:24.068619 8934 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.070887 8917 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c7769e7ca93b413d92d7ec142d7b26b7)
I20251026 03:12:24.070936 8917 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 8 FOLLOWER]: Advancing to term 9
I20251026 03:12:24.071448 8917 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.071535 8678 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 9 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 9. Ops: []
I20251026 03:12:24.071558 8917 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 9 election: Requested vote from peers 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:24.071677 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 9 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:24.071727 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "417cdd6c915c45188264c5da00328e12" candidate_term: 9 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:24.071779 8928 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.071789 8829 raft_consensus.cc:2368] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 417cdd6c915c45188264c5da00328e12 for earlier term 9. Current term is 10.
I20251026 03:12:24.071951 8644 leader_election.cc:400] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 9 election: Vote denied by peer 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853) with higher term. Message: Invalid argument: T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 417cdd6c915c45188264c5da00328e12 for earlier term 9. Current term is 10.
I20251026 03:12:24.071980 8644 leader_election.cc:403] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20251026 03:12:24.072068 8917 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 9 FOLLOWER]: Advancing to term 10
I20251026 03:12:24.072494 8917 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853) with higher term. Message: Invalid argument: T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 417cdd6c915c45188264c5da00328e12 for earlier term 9. Current term is 10.
I20251026 03:12:24.093374 8944 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:24.093422 8944 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 10 FOLLOWER]: Advancing to term 11
I20251026 03:12:24.093883 8944 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.094036 8944 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 11 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
I20251026 03:12:24.094120 8754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 11 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "c7769e7ca93b413d92d7ec142d7b26b7"
I20251026 03:12:24.094173 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "60247008d7194489970aea59d5aa5c7a" candidate_term: 11 candidate_status { last_received { term: 8 index: 16073 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:24.094213 8678 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 10 FOLLOWER]: Advancing to term 11
I20251026 03:12:24.094450 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 11. Ops: []
I20251026 03:12:24.094602 8928 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.094784 8678 raft_consensus.cc:2468] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60247008d7194489970aea59d5aa5c7a in term 11.
I20251026 03:12:24.094940 8792 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a; no voters: c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:24.095007 8944 raft_consensus.cc:2804] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 FOLLOWER]: Leader election won for term 11
I20251026 03:12:24.095052 8944 raft_consensus.cc:697] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 LEADER]: Becoming Leader. State: Replica: 60247008d7194489970aea59d5aa5c7a, State: Running, Role: LEADER
I20251026 03:12:24.095095 8944 consensus_queue.cc:237] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16073, Committed index: 16073, Last appended: 8.16073, Last appended by leader: 16073, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.095402 8574 catalog_manager.cc:5649] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a reported cstate change: term changed from 8 to 11, leader changed from c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66) to 60247008d7194489970aea59d5aa5c7a (127.30.128.67). New cstate: current_term: 11 leader_uuid: "60247008d7194489970aea59d5aa5c7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:24.104189 8678 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 11 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 11. Ops: []
I20251026 03:12:24.104372 8939 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.108088 8829 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 LEADER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 11. Ops: [8.16074-8.16074]
I20251026 03:12:24.108109 8678 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 11 FOLLOWER]: Rejecting Update request from peer c7769e7ca93b413d92d7ec142d7b26b7 for earlier term 8. Current term is 11. Ops: [8.16074-8.16074]
I20251026 03:12:24.108253 8934 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.108316 8934 consensus_queue.cc:1243] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer 60247008d7194489970aea59d5aa5c7a log is divergent from this leader: its last log entry 11.16074 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16073
I20251026 03:12:24.108343 8939 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: INVALID_TERM, Last received: 8.16073, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.108511 8928 raft_consensus.cc:3055] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 8 LEADER]: Stepping down as leader of term 8
I20251026 03:12:24.108538 8928 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c7769e7ca93b413d92d7ec142d7b26b7, State: Running, Role: LEADER
I20251026 03:12:24.108587 8928 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16073, Committed index: 16073, Last appended: 8.16074, Last appended by leader: 16074, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.108652 8928 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 8 FOLLOWER]: Advancing to term 9
I20251026 03:12:24.122505 8678 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 11 FOLLOWER]: Refusing update from remote peer 60247008d7194489970aea59d5aa5c7a: Log matching property violated. Preceding OpId in replica: term: 8 index: 16073. Preceding OpId from leader: term: 11 index: 16074. (index mismatch)
I20251026 03:12:24.122675 8944 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
I20251026 03:12:24.131601 8754 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 9 FOLLOWER]: Advancing to term 11
I20251026 03:12:24.132064 8754 raft_consensus.cc:1275] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 11 FOLLOWER]: Refusing update from remote peer 60247008d7194489970aea59d5aa5c7a: Log matching property violated. Preceding OpId in replica: term: 8 index: 16074. Preceding OpId from leader: term: 11 index: 16074. (term mismatch)
I20251026 03:12:24.132097 8754 pending_rounds.cc:85] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: Aborting all ops after (but not including) 16073
I20251026 03:12:24.132109 8754 pending_rounds.cc:107] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 8.16074
W20251026 03:12:24.132143 8754 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35590: Aborted: Op aborted by new leader
I20251026 03:12:24.132375 8944 consensus_queue.cc:1048] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16074, Last known committed idx: 16073, Time since last communication: 0.000s
W20251026 03:12:24.137699 8668 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39776: Illegal state: replica 417cdd6c915c45188264c5da00328e12 is not leader of this config: current role FOLLOWER
I20251026 03:12:24.173349 8939 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60247008d7194489970aea59d5aa5c7a)
I20251026 03:12:24.173394 8939 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 11 FOLLOWER]: Advancing to term 12
I20251026 03:12:24.173885 8939 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.173993 8939 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 12 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:24.174113 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 12 candidate_status { last_received { term: 11 index: 16075 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:24.174134 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 12 candidate_status { last_received { term: 11 index: 16075 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:24.174296 8717 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7769e7ca93b413d92d7ec142d7b26b7; no voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a
I20251026 03:12:24.174366 8939 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251026 03:12:24.176266 8754 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 12 FOLLOWER]: Rejecting Update request from peer 60247008d7194489970aea59d5aa5c7a for earlier term 11. Current term is 12. Ops: []
I20251026 03:12:24.176448 8946 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: INVALID_TERM, Last received: 11.16075, Next index: 16076, Last known committed idx: 16075, Time since last communication: 0.000s
I20251026 03:12:24.201911 8754 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 12 FOLLOWER]: Rejecting Update request from peer 60247008d7194489970aea59d5aa5c7a for earlier term 11. Current term is 12. Ops: []
I20251026 03:12:24.202059 8940 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: INVALID_TERM, Last received: 11.16075, Next index: 16076, Last known committed idx: 16075, Time since last communication: 0.000s
I20251026 03:12:24.218730 8939 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:24.218771 8939 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 12 FOLLOWER]: Advancing to term 13
I20251026 03:12:24.219990 8939 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.220101 8939 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 13 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), 60247008d7194489970aea59d5aa5c7a (127.30.128.67:37853)
I20251026 03:12:24.220244 8829 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 13 candidate_status { last_received { term: 11 index: 16075 } } ignore_live_leader: false dest_uuid: "60247008d7194489970aea59d5aa5c7a"
I20251026 03:12:24.220299 8678 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3345c67fff444b66ab9a03779e1022cb" candidate_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" candidate_term: 13 candidate_status { last_received { term: 11 index: 16075 } } ignore_live_leader: false dest_uuid: "417cdd6c915c45188264c5da00328e12"
I20251026 03:12:24.220468 8717 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7769e7ca93b413d92d7ec142d7b26b7; no voters: 417cdd6c915c45188264c5da00328e12, 60247008d7194489970aea59d5aa5c7a
I20251026 03:12:24.220551 8939 raft_consensus.cc:2749] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20251026 03:12:24.235335 8754 raft_consensus.cc:1240] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 13 FOLLOWER]: Rejecting Update request from peer 60247008d7194489970aea59d5aa5c7a for earlier term 11. Current term is 13. Ops: []
I20251026 03:12:24.235497 8944 consensus_queue.cc:1059] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 }, Status: INVALID_TERM, Last received: 11.16075, Next index: 16076, Last known committed idx: 16075, Time since last communication: 0.000s
I20251026 03:12:24.253067 31233 tablet_server.cc:178] TabletServer@127.30.128.65:0 shutting down...
I20251026 03:12:24.253866 8946 raft_consensus.cc:3055] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 LEADER]: Stepping down as leader of term 11
I20251026 03:12:24.253916 8946 raft_consensus.cc:740] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60247008d7194489970aea59d5aa5c7a, State: Running, Role: LEADER
I20251026 03:12:24.253966 8946 consensus_queue.cc:260] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16075, Committed index: 16075, Last appended: 11.16075, Last appended by leader: 16075, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.254052 8946 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 11 FOLLOWER]: Advancing to term 12
I20251026 03:12:24.255303 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:24.255379 31233 tablet_replica.cc:333] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12: stopping tablet replica
I20251026 03:12:24.255425 31233 raft_consensus.cc:2243] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 11 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:24.255462 31233 raft_consensus.cc:2272] T 3345c67fff444b66ab9a03779e1022cb P 417cdd6c915c45188264c5da00328e12 [term 11 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:24.269418 31233 tablet_server.cc:195] TabletServer@127.30.128.65:0 shutdown complete.
I20251026 03:12:24.270185 31233 tablet_server.cc:178] TabletServer@127.30.128.66:0 shutting down...
I20251026 03:12:24.272296 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:24.272379 31233 tablet_replica.cc:333] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7: stopping tablet replica
I20251026 03:12:24.272421 31233 raft_consensus.cc:2243] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 13 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:24.277048 31233 raft_consensus.cc:2272] T 3345c67fff444b66ab9a03779e1022cb P c7769e7ca93b413d92d7ec142d7b26b7 [term 13 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:24.287783 31233 tablet_server.cc:195] TabletServer@127.30.128.66:0 shutdown complete.
I20251026 03:12:24.288393 31233 tablet_server.cc:178] TabletServer@127.30.128.67:0 shutting down...
I20251026 03:12:24.290534 8944 raft_consensus.cc:493] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:24.290584 8944 raft_consensus.cc:3060] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 12 FOLLOWER]: Advancing to term 13
I20251026 03:12:24.290635 31233 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 03:12:24.290745 31233 tablet_replica.cc:333] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a: stopping tablet replica
I20251026 03:12:24.291095 8944 raft_consensus.cc:515] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "417cdd6c915c45188264c5da00328e12" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 43855 } } peers { permanent_uuid: "60247008d7194489970aea59d5aa5c7a" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 37853 } } peers { permanent_uuid: "c7769e7ca93b413d92d7ec142d7b26b7" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 41919 } }
I20251026 03:12:24.291177 31233 raft_consensus.cc:2243] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 13 FOLLOWER]: Raft consensus shutting down.
I20251026 03:12:24.291201 8944 leader_election.cc:290] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 13 election: Requested vote from peers 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855), c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919)
W20251026 03:12:24.291476 8795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.128.66:41919: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20251026 03:12:24.291762 8795 leader_election.cc:336] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer c7769e7ca93b413d92d7ec142d7b26b7 (127.30.128.66:41919): Network error: Client connection negotiation failed: client connection to 127.30.128.66:41919: connect: Connection refused (error 111)
W20251026 03:12:24.291812 8792 leader_election.cc:336] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 417cdd6c915c45188264c5da00328e12 (127.30.128.65:43855): Network error: Client connection negotiation failed: client connection to 127.30.128.65:43855: connect: Connection refused (error 111)
I20251026 03:12:24.291857 8792 leader_election.cc:304] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60247008d7194489970aea59d5aa5c7a; no voters: 417cdd6c915c45188264c5da00328e12, c7769e7ca93b413d92d7ec142d7b26b7
I20251026 03:12:24.291918 8944 raft_consensus.cc:2706] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 13 FOLLOWER]: Received election callback for term 13 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20251026 03:12:24.311146 31233 raft_consensus.cc:2272] T 3345c67fff444b66ab9a03779e1022cb P 60247008d7194489970aea59d5aa5c7a [term 13 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:24.323225 31233 tablet_server.cc:195] TabletServer@127.30.128.67:0 shutdown complete.
I20251026 03:12:24.323971 31233 master.cc:561] Master@127.30.128.126:32851 shutting down...
I20251026 03:12:24.325603 31233 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 1 LEADER]: Raft consensus shutting down.
I20251026 03:12:24.325659 31233 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 03:12:24.325673 31233 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9fa7c9b520aa416896fa8615540ab310: stopping tablet replica
I20251026 03:12:24.337095 31233 master.cc:583] Master@127.30.128.126:32851 shutdown complete.
[ OK ] TxnParticipantElectionStormITest.TestTxnSystemClientRetriesThroughStorm (10862 ms)
[----------] 2 tests from TxnParticipantElectionStormITest (27629 ms total)
[----------] 2 tests from ShouldGCWals/ParticipantCopyITest
[ RUN ] ShouldGCWals/ParticipantCopyITest.TestCopyParticipantOps/0
I20251026 03:12:24.355445 31233 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.128.126:37407
I20251026 03:12:24.355552 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:24.356698 8951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:24.356732 8950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:24.356815 8953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:24.356914 31233 server_base.cc:1047] running on GCE node
I20251026 03:12:24.357098 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:24.357169 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:24.357187 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448344357187 us; error 0 us; skew 500 ppm
I20251026 03:12:24.362308 31233 webserver.cc:492] Webserver started at http://127.30.128.126:41521/ using document root <none> and password file <none>
I20251026 03:12:24.362389 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:24.362430 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:24.362478 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:24.362695 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/master-0-root/instance:
uuid: "d41e1a2534d54c36ba74a69211b5a261"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.363602 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.364053 8958 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.364173 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.364223 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/master-0-root
uuid: "d41e1a2534d54c36ba74a69211b5a261"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.364260 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:24.373245 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:24.373364 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:24.375475 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.126:37407
I20251026 03:12:24.377727 9010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.126:37407 every 8 connection(s)
I20251026 03:12:24.378022 9011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:24.378803 9011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261: Bootstrap starting.
I20251026 03:12:24.379029 9011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:24.379526 9011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261: No bootstrap required, opened a new log
I20251026 03:12:24.379645 9011 raft_consensus.cc:359] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e1a2534d54c36ba74a69211b5a261" member_type: VOTER }
I20251026 03:12:24.379694 9011 raft_consensus.cc:385] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:24.379709 9011 raft_consensus.cc:740] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41e1a2534d54c36ba74a69211b5a261, State: Initialized, Role: FOLLOWER
I20251026 03:12:24.379755 9011 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e1a2534d54c36ba74a69211b5a261" member_type: VOTER }
I20251026 03:12:24.379803 9011 raft_consensus.cc:399] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 03:12:24.379837 9011 raft_consensus.cc:493] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 03:12:24.379858 9011 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:24.380296 9011 raft_consensus.cc:515] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e1a2534d54c36ba74a69211b5a261" member_type: VOTER }
I20251026 03:12:24.380347 9011 leader_election.cc:304] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41e1a2534d54c36ba74a69211b5a261; no voters:
I20251026 03:12:24.380450 9011 leader_election.cc:290] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 03:12:24.380512 9014 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:12:24.380591 9011 sys_catalog.cc:565] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 03:12:24.380630 9014 raft_consensus.cc:697] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [term 1 LEADER]: Becoming Leader. State: Replica: d41e1a2534d54c36ba74a69211b5a261, State: Running, Role: LEADER
I20251026 03:12:24.380671 9014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e1a2534d54c36ba74a69211b5a261" member_type: VOTER }
I20251026 03:12:24.380824 9014 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41e1a2534d54c36ba74a69211b5a261. Latest consensus state: current_term: 1 leader_uuid: "d41e1a2534d54c36ba74a69211b5a261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e1a2534d54c36ba74a69211b5a261" member_type: VOTER } }
I20251026 03:12:24.380885 9014 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [sys.catalog]: This master's current role is: LEADER
I20251026 03:12:24.380849 9015 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41e1a2534d54c36ba74a69211b5a261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e1a2534d54c36ba74a69211b5a261" member_type: VOTER } }
I20251026 03:12:24.381028 9015 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261 [sys.catalog]: This master's current role is: LEADER
I20251026 03:12:24.381088 9019 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 03:12:24.381217 9019 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 03:12:24.381682 9019 catalog_manager.cc:1357] Generated new cluster ID: b5b035f0569d4cfd9c3eb55312fe479e
I20251026 03:12:24.381709 31233 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 03:12:24.381726 9019 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 03:12:24.403172 9019 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 03:12:24.403501 9019 catalog_manager.cc:1514] Loading token signing keys...
I20251026 03:12:24.410287 9019 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d41e1a2534d54c36ba74a69211b5a261: Generated new TSK 0
I20251026 03:12:24.410373 9019 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 03:12:24.413465 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:24.414579 9032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:24.414625 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:24.414564 9035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:24.414564 9033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:24.414801 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:24.414832 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:24.414858 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448344414859 us; error 0 us; skew 500 ppm
I20251026 03:12:24.415257 31233 webserver.cc:492] Webserver started at http://127.30.128.65:43751/ using document root <none> and password file <none>
I20251026 03:12:24.415311 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:24.415338 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:24.415364 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:24.415544 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/instance:
uuid: "a74cdf9569d5402fa3b5bcc40875eb2f"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.416191 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251026 03:12:24.416512 9040 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.416594 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.416636 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root
uuid: "a74cdf9569d5402fa3b5bcc40875eb2f"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.416671 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:24.426609 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:24.426739 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:24.426844 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:24.426995 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:12:24.427016 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.427059 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:12:24.427083 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.429545 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.65:45735
I20251026 03:12:24.430094 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 03:12:24.430473 9103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.65:45735 every 8 connection(s)
W20251026 03:12:24.431071 9106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:24.431255 9107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:24.431402 31233 server_base.cc:1047] running on GCE node
W20251026 03:12:24.431483 9109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:24.431602 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:24.431630 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:24.431644 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448344431644 us; error 0 us; skew 500 ppm
I20251026 03:12:24.432197 31233 webserver.cc:492] Webserver started at http://127.30.128.66:43581/ using document root <none> and password file <none>
I20251026 03:12:24.432266 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:24.432305 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:24.432353 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:24.432556 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/instance:
uuid: "e46fbbdc3a05494998b24380ff3e0244"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.433379 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.433733 9104 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37407
I20251026 03:12:24.433810 9104 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:24.433897 9116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.433910 9104 heartbeater.cc:507] Master 127.30.128.126:37407 requested a full tablet report, sending...
I20251026 03:12:24.434072 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.434113 8975 ts_manager.cc:194] Registered new tserver with Master: a74cdf9569d5402fa3b5bcc40875eb2f (127.30.128.65:45735)
I20251026 03:12:24.434123 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root
uuid: "e46fbbdc3a05494998b24380ff3e0244"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.434182 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:24.434736 8975 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44972
I20251026 03:12:24.445996 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:24.446132 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:24.446244 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:24.446400 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:12:24.446424 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.446453 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:12:24.446478 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.449051 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.66:44841
I20251026 03:12:24.449110 9179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.66:44841 every 8 connection(s)
I20251026 03:12:24.449397 31233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 03:12:24.450412 9184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:24.450397 9183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 03:12:24.450712 9186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 03:12:24.450886 31233 server_base.cc:1047] running on GCE node
I20251026 03:12:24.450963 31233 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 03:12:24.450994 31233 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 03:12:24.451010 31233 hybrid_clock.cc:648] HybridClock initialized: now 1761448344451010 us; error 0 us; skew 500 ppm
I20251026 03:12:24.451474 31233 webserver.cc:492] Webserver started at http://127.30.128.67:45321/ using document root <none> and password file <none>
I20251026 03:12:24.451552 31233 fs_manager.cc:362] Metadata directory not provided
I20251026 03:12:24.451607 31233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 03:12:24.451659 31233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 03:12:24.451903 31233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/instance:
uuid: "f11d9d7d1c1446a9a10998d50d86c657"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.452400 9180 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37407
I20251026 03:12:24.452452 9180 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:24.452541 9180 heartbeater.cc:507] Master 127.30.128.126:37407 requested a full tablet report, sending...
I20251026 03:12:24.452760 8975 ts_manager.cc:194] Registered new tserver with Master: e46fbbdc3a05494998b24380ff3e0244 (127.30.128.66:44841)
I20251026 03:12:24.452773 31233 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.453099 8975 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44974
I20251026 03:12:24.453194 9191 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.453318 31233 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.453359 31233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root
uuid: "f11d9d7d1c1446a9a10998d50d86c657"
format_stamp: "Formatted at 2025-10-26 03:12:24 on dist-test-slave-51kf"
I20251026 03:12:24.453408 31233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 03:12:24.462179 31233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 03:12:24.462314 31233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 03:12:24.462419 31233 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 03:12:24.462553 31233 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 03:12:24.462575 31233 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.462591 31233 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 03:12:24.462623 31233 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.465554 31233 rpc_server.cc:307] RPC server started. Bound to: 127.30.128.67:36573
I20251026 03:12:24.465595 9254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.128.67:36573 every 8 connection(s)
I20251026 03:12:24.467867 9255 heartbeater.cc:344] Connected to a master server at 127.30.128.126:37407
I20251026 03:12:24.467911 9255 heartbeater.cc:461] Registering TS with master...
I20251026 03:12:24.467994 9255 heartbeater.cc:507] Master 127.30.128.126:37407 requested a full tablet report, sending...
I20251026 03:12:24.468169 8975 ts_manager.cc:194] Registered new tserver with Master: f11d9d7d1c1446a9a10998d50d86c657 (127.30.128.67:36573)
I20251026 03:12:24.468417 8975 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44986
I20251026 03:12:24.468950 31233 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003261636s
I20251026 03:12:24.469236 31233 test_util.cc:276] Using random seed: 652001012
I20251026 03:12:24.472873 8975 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44990:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 03:12:24.472975 8975 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 03:12:24.476830 9068 tablet_service.cc:1505] Processing CreateTablet for tablet 6faa7d3ea6644b65aca9c9fe97dbc238 (DEFAULT_TABLE table=test-workload [id=5b843604bd5c4da08a42e9657e0378fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:12:24.476961 9219 tablet_service.cc:1505] Processing CreateTablet for tablet 6faa7d3ea6644b65aca9c9fe97dbc238 (DEFAULT_TABLE table=test-workload [id=5b843604bd5c4da08a42e9657e0378fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:12:24.477005 9144 tablet_service.cc:1505] Processing CreateTablet for tablet 6faa7d3ea6644b65aca9c9fe97dbc238 (DEFAULT_TABLE table=test-workload [id=5b843604bd5c4da08a42e9657e0378fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 03:12:24.477042 9219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faa7d3ea6644b65aca9c9fe97dbc238. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:24.477059 9144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faa7d3ea6644b65aca9c9fe97dbc238. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:24.476953 9068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faa7d3ea6644b65aca9c9fe97dbc238. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 03:12:24.478205 9270 tablet_bootstrap.cc:492] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Bootstrap starting.
I20251026 03:12:24.478523 9270 tablet_bootstrap.cc:654] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:24.478756 9272 tablet_bootstrap.cc:492] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Bootstrap starting.
I20251026 03:12:24.478811 9271 tablet_bootstrap.cc:492] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Bootstrap starting.
I20251026 03:12:24.479146 9272 tablet_bootstrap.cc:654] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:24.479146 9271 tablet_bootstrap.cc:654] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Neither blocks nor log segments found. Creating new log.
I20251026 03:12:24.479303 9270 tablet_bootstrap.cc:492] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: No bootstrap required, opened a new log
I20251026 03:12:24.479351 9270 ts_tablet_manager.cc:1403] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.479506 9270 raft_consensus.cc:359] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.479647 9270 raft_consensus.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:24.479682 9270 raft_consensus.cc:740] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11d9d7d1c1446a9a10998d50d86c657, State: Initialized, Role: FOLLOWER
I20251026 03:12:24.479759 9272 tablet_bootstrap.cc:492] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: No bootstrap required, opened a new log
I20251026 03:12:24.479753 9270 consensus_queue.cc:260] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.479835 9272 ts_tablet_manager.cc:1403] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.479847 9271 tablet_bootstrap.cc:492] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: No bootstrap required, opened a new log
I20251026 03:12:24.479892 9271 ts_tablet_manager.cc:1403] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.479944 9270 ts_tablet_manager.cc:1434] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 03:12:24.480016 9255 heartbeater.cc:499] Master 127.30.128.126:37407 was elected leader, sending a full tablet report...
I20251026 03:12:24.479979 9272 raft_consensus.cc:359] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.480036 9271 raft_consensus.cc:359] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.480070 9272 raft_consensus.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:24.480084 9271 raft_consensus.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 03:12:24.480103 9272 raft_consensus.cc:740] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74cdf9569d5402fa3b5bcc40875eb2f, State: Initialized, Role: FOLLOWER
I20251026 03:12:24.480108 9271 raft_consensus.cc:740] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e46fbbdc3a05494998b24380ff3e0244, State: Initialized, Role: FOLLOWER
I20251026 03:12:24.480159 9272 consensus_queue.cc:260] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.480202 9271 consensus_queue.cc:260] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.480314 9272 ts_tablet_manager.cc:1434] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251026 03:12:24.480361 9271 ts_tablet_manager.cc:1434] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251026 03:12:24.480402 9104 heartbeater.cc:499] Master 127.30.128.126:37407 was elected leader, sending a full tablet report...
I20251026 03:12:24.480418 9180 heartbeater.cc:499] Master 127.30.128.126:37407 was elected leader, sending a full tablet report...
W20251026 03:12:24.480988 9105 tablet.cc:2378] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 03:12:24.683791 9277 raft_consensus.cc:493] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 03:12:24.683894 9277 raft_consensus.cc:515] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.684054 9277 leader_election.cc:290] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11d9d7d1c1446a9a10998d50d86c657 (127.30.128.67:36573), e46fbbdc3a05494998b24380ff3e0244 (127.30.128.66:44841)
I20251026 03:12:24.686276 9229 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6faa7d3ea6644b65aca9c9fe97dbc238" candidate_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11d9d7d1c1446a9a10998d50d86c657" is_pre_election: true
I20251026 03:12:24.686340 9154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6faa7d3ea6644b65aca9c9fe97dbc238" candidate_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e46fbbdc3a05494998b24380ff3e0244" is_pre_election: true
I20251026 03:12:24.686380 9229 raft_consensus.cc:2468] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74cdf9569d5402fa3b5bcc40875eb2f in term 0.
I20251026 03:12:24.686398 9154 raft_consensus.cc:2468] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74cdf9569d5402fa3b5bcc40875eb2f in term 0.
I20251026 03:12:24.686517 9044 leader_election.cc:304] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74cdf9569d5402fa3b5bcc40875eb2f, f11d9d7d1c1446a9a10998d50d86c657; no voters:
I20251026 03:12:24.686592 9277 raft_consensus.cc:2804] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 03:12:24.686641 9277 raft_consensus.cc:493] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 03:12:24.686668 9277 raft_consensus.cc:3060] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:24.687229 9277 raft_consensus.cc:515] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.687342 9277 leader_election.cc:290] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [CANDIDATE]: Term 1 election: Requested vote from peers f11d9d7d1c1446a9a10998d50d86c657 (127.30.128.67:36573), e46fbbdc3a05494998b24380ff3e0244 (127.30.128.66:44841)
I20251026 03:12:24.687467 9154 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6faa7d3ea6644b65aca9c9fe97dbc238" candidate_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e46fbbdc3a05494998b24380ff3e0244"
I20251026 03:12:24.687474 9229 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6faa7d3ea6644b65aca9c9fe97dbc238" candidate_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11d9d7d1c1446a9a10998d50d86c657"
I20251026 03:12:24.687523 9154 raft_consensus.cc:3060] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:24.687542 9229 raft_consensus.cc:3060] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 0 FOLLOWER]: Advancing to term 1
I20251026 03:12:24.688149 9154 raft_consensus.cc:2468] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74cdf9569d5402fa3b5bcc40875eb2f in term 1.
I20251026 03:12:24.688154 9229 raft_consensus.cc:2468] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74cdf9569d5402fa3b5bcc40875eb2f in term 1.
I20251026 03:12:24.688305 9042 leader_election.cc:304] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74cdf9569d5402fa3b5bcc40875eb2f, e46fbbdc3a05494998b24380ff3e0244; no voters:
I20251026 03:12:24.688381 9277 raft_consensus.cc:2804] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 1 FOLLOWER]: Leader election won for term 1
I20251026 03:12:24.688510 9277 raft_consensus.cc:697] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [term 1 LEADER]: Becoming Leader. State: Replica: a74cdf9569d5402fa3b5bcc40875eb2f, State: Running, Role: LEADER
I20251026 03:12:24.688555 9277 consensus_queue.cc:237] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } }
I20251026 03:12:24.688999 8975 catalog_manager.cc:5649] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f reported cstate change: term changed from 0 to 1, leader changed from <none> to a74cdf9569d5402fa3b5bcc40875eb2f (127.30.128.65). New cstate: current_term: 1 leader_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a74cdf9569d5402fa3b5bcc40875eb2f" member_type: VOTER last_known_addr { host: "127.30.128.65" port: 45735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 } health_report { overall_health: UNKNOWN } } }
I20251026 03:12:24.710105 9229 raft_consensus.cc:1275] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657 [term 1 FOLLOWER]: Refusing update from remote peer a74cdf9569d5402fa3b5bcc40875eb2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:12:24.710105 9154 raft_consensus.cc:1275] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244 [term 1 FOLLOWER]: Refusing update from remote peer a74cdf9569d5402fa3b5bcc40875eb2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 03:12:24.710311 9280 consensus_queue.cc:1048] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e46fbbdc3a05494998b24380ff3e0244" member_type: VOTER last_known_addr { host: "127.30.128.66" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:12:24.710403 9277 consensus_queue.cc:1048] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11d9d7d1c1446a9a10998d50d86c657" member_type: VOTER last_known_addr { host: "127.30.128.67" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 03:12:24.712221 9289 mvcc.cc:204] Tried to move back new op lower bound from 7214892419932905472 to 7214892419844771840. Current Snapshot: MvccSnapshot[applied={T|T < 7214892419931492352}]
I20251026 03:12:24.819895 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:24.823997 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:24.829329 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:24.837112 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":271136,"cfile_init":1,"compiler_manager_pool.queue_time_us":1718,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":689,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":523,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400,"thread_start_us":95,"threads_started":1}
I20251026 03:12:24.837378 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 86531 bytes on disk
I20251026 03:12:24.837561 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:24.850464 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.021s user 0.009s sys 0.004s Metrics: {"bytes_written":279182,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":464,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20251026 03:12:24.850703 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 89309 bytes on disk
I20251026 03:12:24.850821 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.007s sys 0.007s Metrics: {"bytes_written":279182,"cfile_init":1,"dirs.queue_time_us":134,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":528,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20251026 03:12:24.850978 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 89309 bytes on disk
I20251026 03:12:24.851166 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:24.851503 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:24.989531 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:24.992794 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:24.992872 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.007987 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.008s sys 0.005s Metrics: {"bytes_written":284422,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":762,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":683,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20251026 03:12:25.008212 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014707
I20251026 03:12:25.009490 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.017s user 0.013s sys 0.000s Metrics: {"bytes_written":281570,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":100,"dirs.run_wall_time_us":549,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20251026 03:12:25.012100 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 90301 bytes on disk
I20251026 03:12:25.012948 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.020s user 0.006s sys 0.006s Metrics: {"bytes_written":281570,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":479,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":530,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800,"spinlock_wait_cycles":191872}
I20251026 03:12:25.014364 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1258,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20251026 03:12:25.014472 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014706
I20251026 03:12:25.014557 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014706
I20251026 03:12:25.063293 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.048s user 0.022s sys 0.006s Metrics: {"bytes_written":556854,"cfile_cache_hit":12,"cfile_cache_hit_bytes":428301,"cfile_cache_miss":40,"cfile_cache_miss_bytes":951999,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":642,"drs_written":1,"lbm_read_time_us":622,"lbm_reads_lt_1ms":64,"lbm_write_time_us":1141,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":298859,"rows_written":19500}
I20251026 03:12:25.066414 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 184894 bytes on disk
I20251026 03:12:25.067490 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.059s user 0.022s sys 0.005s Metrics: {"bytes_written":551552,"cfile_cache_hit":12,"cfile_cache_hit_bytes":422535,"cfile_cache_miss":40,"cfile_cache_miss_bytes":943623,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":542,"drs_written":1,"lbm_read_time_us":603,"lbm_reads_lt_1ms":64,"lbm_write_time_us":1028,"lbm_writes_lt_1ms":43,"num_input_rowsets":2,"peak_mem_usage":295809,"rows_written":19300,"thread_start_us":1217,"threads_started":5}
I20251026 03:12:25.067625 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 182875 bytes on disk
I20251026 03:12:25.067726 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.068420 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.107980 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.093s user 0.024s sys 0.003s Metrics: {"bytes_written":556854,"cfile_cache_hit":12,"cfile_cache_hit_bytes":428301,"cfile_cache_miss":40,"cfile_cache_miss_bytes":951999,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":733,"drs_written":1,"lbm_read_time_us":1358,"lbm_reads_lt_1ms":60,"lbm_write_time_us":58753,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":41,"num_input_rowsets":2,"peak_mem_usage":298859,"rows_written":19500,"thread_start_us":263,"threads_started":5}
I20251026 03:12:25.108244 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 184894 bytes on disk
I20251026 03:12:25.108436 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.148670 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.148939 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.149531 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.169687 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.020s user 0.008s sys 0.004s Metrics: {"bytes_written":280313,"cfile_init":1,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":589,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20251026 03:12:25.169914 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014677
I20251026 03:12:25.171365 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.022s user 0.013s sys 0.000s Metrics: {"bytes_written":279098,"cfile_init":1,"dirs.queue_time_us":2393,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":514,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20251026 03:12:25.171473 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1048830 bytes of WAL
I20251026 03:12:25.171530 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:25.171561 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000001 (ops 1-579)
I20251026 03:12:25.171944 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:25.172025 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014678
I20251026 03:12:25.177160 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.012s sys 0.000s Metrics: {"bytes_written":283140,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":509,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20251026 03:12:25.177305 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014678
I20251026 03:12:25.215945 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.039s user 0.036s sys 0.000s Metrics: {"bytes_written":832330,"cfile_cache_hit":12,"cfile_cache_hit_bytes":562442,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1508330,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":121,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":596,"drs_written":1,"lbm_read_time_us":798,"lbm_reads_lt_1ms":74,"lbm_write_time_us":1499,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":446766,"rows_written":29200}
I20251026 03:12:25.216116 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 282533 bytes on disk
I20251026 03:12:25.216209 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.230903 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.061s user 0.035s sys 0.000s Metrics: {"bytes_written":833757,"cfile_cache_hit":12,"cfile_cache_hit_bytes":563407,"cfile_cache_miss":50,"cfile_cache_miss_bytes":1510837,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":503,"drs_written":1,"lbm_read_time_us":1167,"lbm_reads_lt_1ms":74,"lbm_write_time_us":1860,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":445876,"rows_written":29250}
I20251026 03:12:25.231032 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 283043 bytes on disk
I20251026 03:12:25.231119 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.234623 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.057s user 0.028s sys 0.008s Metrics: {"bytes_written":831066,"cfile_cache_hit":10,"cfile_cache_hit_bytes":448178,"cfile_cache_miss":52,"cfile_cache_miss_bytes":1618826,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":446,"drs_written":1,"lbm_read_time_us":962,"lbm_reads_lt_1ms":76,"lbm_write_time_us":1476,"lbm_writes_lt_1ms":55,"num_input_rowsets":2,"peak_mem_usage":444351,"rows_written":29150}
I20251026 03:12:25.234956 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 282087 bytes on disk
I20251026 03:12:25.235260 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.291819 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.295996 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.297183 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.326222 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.029s user 0.013s sys 0.000s Metrics: {"bytes_written":286838,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":426,"drs_written":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4,"lbm_write_time_us":929,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":384}
I20251026 03:12:25.326349 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 92305 bytes on disk
I20251026 03:12:25.327559 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.035s user 0.012s sys 0.001s Metrics: {"bytes_written":276377,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":735,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20251026 03:12:25.327668 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1061804 bytes of WAL
I20251026 03:12:25.328126 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:25.328164 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000001 (ops 1-634)
I20251026 03:12:25.328481 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:25.328562 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 88617 bytes on disk
I20251026 03:12:25.328629 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":858,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.328701 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014498
I20251026 03:12:25.330040 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.330209 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014502
I20251026 03:12:25.331638 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.035s user 0.012s sys 0.001s Metrics: {"bytes_written":285602,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":470,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":617,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9950}
I20251026 03:12:25.331751 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1124248 bytes of WAL
I20251026 03:12:25.331816 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:25.331840 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000001 (ops 1-671)
I20251026 03:12:25.332898 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:25.332991 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 91836 bytes on disk
I20251026 03:12:25.334754 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.334853 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014499
I20251026 03:12:25.387157 8964 maintenance_manager.cc:419] P d41e1a2534d54c36ba74a69211b5a261: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.003577
I20251026 03:12:25.388980 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.054s user 0.041s sys 0.005s Metrics: {"bytes_written":1107902,"cfile_cache_hit":10,"cfile_cache_hit_bytes":450198,"cfile_cache_miss":64,"cfile_cache_miss_bytes":2331696,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":557,"drs_written":1,"lbm_read_time_us":1233,"lbm_reads_lt_1ms":84,"lbm_write_time_us":1862,"lbm_writes_lt_1ms":65,"num_input_rowsets":2,"peak_mem_usage":594416,"rows_written":39100}
I20251026 03:12:25.389106 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 383547 bytes on disk
I20251026 03:12:25.389940 8963 maintenance_manager.cc:643] P d41e1a2534d54c36ba74a69211b5a261: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":6633,"cfile_init":1,"dirs.queue_time_us":85,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":166,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":5,"thread_start_us":53,"threads_started":1}
I20251026 03:12:25.390071 8964 maintenance_manager.cc:419] P d41e1a2534d54c36ba74a69211b5a261: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 487 bytes on disk
I20251026 03:12:25.390838 8963 maintenance_manager.cc:643] P d41e1a2534d54c36ba74a69211b5a261: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.390990 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.418397 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.090s user 0.046s sys 0.000s Metrics: {"bytes_written":1110461,"cfile_cache_hit":11,"cfile_cache_hit_bytes":536321,"cfile_cache_miss":63,"cfile_cache_miss_bytes":2252409,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":79,"dirs.run_wall_time_us":474,"drs_written":1,"lbm_read_time_us":1085,"lbm_reads_lt_1ms":83,"lbm_write_time_us":4098,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":64,"num_input_rowsets":2,"peak_mem_usage":599241,"rows_written":39200}
I20251026 03:12:25.418475 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.088s user 0.041s sys 0.005s Metrics: {"bytes_written":1101427,"cfile_cache_hit":10,"cfile_cache_hit_bytes":443456,"cfile_cache_miss":64,"cfile_cache_miss_bytes":2320850,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":559,"drs_written":1,"lbm_read_time_us":1048,"lbm_reads_lt_1ms":84,"lbm_write_time_us":6905,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":62,"num_input_rowsets":2,"peak_mem_usage":592256,"rows_written":38850}
I20251026 03:12:25.418694 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 384528 bytes on disk
I20251026 03:12:25.418741 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 380588 bytes on disk
I20251026 03:12:25.419239 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.419516 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.459820 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.460083 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.461725 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.473330 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.011s user 0.006s sys 0.004s Metrics: {"bytes_written":275045,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":609,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":475,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20251026 03:12:25.473451 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 88023 bytes on disk
I20251026 03:12:25.473549 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.473634 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014360
I20251026 03:12:25.475318 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":272594,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":567,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20251026 03:12:25.475430 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014359
I20251026 03:12:25.478829 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.019s user 0.010s sys 0.002s Metrics: {"bytes_written":281582,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":438,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20251026 03:12:25.479010 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 90312 bytes on disk
I20251026 03:12:25.480278 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.480901 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014361
I20251026 03:12:25.547793 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.067s user 0.047s sys 0.008s Metrics: {"bytes_written":1376078,"cfile_cache_hit":12,"cfile_cache_hit_bytes":670723,"cfile_cache_miss":70,"cfile_cache_miss_bytes":2796633,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":816,"drs_written":1,"lbm_read_time_us":1362,"lbm_reads_lt_1ms":90,"lbm_write_time_us":2168,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":741688,"rows_written":48650}
I20251026 03:12:25.548070 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 481477 bytes on disk
I20251026 03:12:25.548210 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.552058 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.078s user 0.055s sys 0.000s Metrics: {"bytes_written":1376078,"cfile_cache_hit":13,"cfile_cache_hit_bytes":697109,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2771453,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":854,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":579,"drs_written":1,"lbm_read_time_us":1819,"lbm_reads_lt_1ms":91,"lbm_write_time_us":3195,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":741681,"rows_written":48650}
I20251026 03:12:25.552196 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 481477 bytes on disk
I20251026 03:12:25.552331 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.552536 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.077s user 0.048s sys 0.011s Metrics: {"bytes_written":1376078,"cfile_cache_hit":13,"cfile_cache_hit_bytes":695443,"cfile_cache_miss":71,"cfile_cache_miss_bytes":2773255,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":450,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":399,"drs_written":1,"lbm_read_time_us":1840,"lbm_reads_lt_1ms":95,"lbm_write_time_us":3211,"lbm_writes_lt_1ms":76,"num_input_rowsets":2,"peak_mem_usage":741681,"rows_written":48650}
I20251026 03:12:25.552654 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 481477 bytes on disk
I20251026 03:12:25.552906 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.639410 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.643384 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.644616 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.648785 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.009s user 0.008s sys 0.000s Metrics: {"bytes_written":276415,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":630,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":476,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20251026 03:12:25.648909 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014253
I20251026 03:12:25.655767 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":280208,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":565,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20251026 03:12:25.655896 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 89741 bytes on disk
I20251026 03:12:25.656002 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.656090 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014251
I20251026 03:12:25.685881 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.041s user 0.013s sys 0.000s Metrics: {"bytes_written":280208,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":636,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20251026 03:12:25.686023 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1047900 bytes of WAL
I20251026 03:12:25.686403 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:25.686731 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000002 (ops 580-1156)
I20251026 03:12:25.687265 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:25.687362 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 89741 bytes on disk
I20251026 03:12:25.687450 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.687878 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014251
I20251026 03:12:25.735250 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.079s user 0.058s sys 0.007s Metrics: {"bytes_written":1650076,"cfile_cache_hit":12,"cfile_cache_hit_bytes":721821,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3443299,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":119,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":519,"drs_written":1,"lbm_read_time_us":1558,"lbm_reads_lt_1ms":102,"lbm_write_time_us":3559,"lbm_writes_lt_1ms":88,"num_input_rowsets":2,"peak_mem_usage":888698,"rows_written":58400}
I20251026 03:12:25.735505 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 581958 bytes on disk
I20251026 03:12:25.735661 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.752797 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.104s user 0.055s sys 0.006s Metrics: {"bytes_written":1646380,"cfile_cache_hit":12,"cfile_cache_hit_bytes":718931,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3435857,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":713,"drs_written":1,"lbm_read_time_us":2269,"lbm_reads_lt_1ms":106,"lbm_write_time_us":3815,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":87,"num_input_rowsets":2,"peak_mem_usage":888063,"rows_written":58250}
I20251026 03:12:25.752955 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 580163 bytes on disk
I20251026 03:12:25.753468 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.796356 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.108s user 0.047s sys 0.016s Metrics: {"bytes_written":1650076,"cfile_cache_hit":12,"cfile_cache_hit_bytes":721821,"cfile_cache_miss":82,"cfile_cache_miss_bytes":3443299,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":523,"drs_written":1,"lbm_read_time_us":2398,"lbm_reads_lt_1ms":102,"lbm_write_time_us":3275,"lbm_writes_lt_1ms":88,"num_input_rowsets":2,"peak_mem_usage":888698,"rows_written":58400}
I20251026 03:12:25.796546 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 581958 bytes on disk
I20251026 03:12:25.797097 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.814425 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.817379 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.817482 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:25.839617 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.024s user 0.011s sys 0.002s Metrics: {"bytes_written":281650,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":569,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1512,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20251026 03:12:25.839870 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1050403 bytes of WAL
I20251026 03:12:25.841100 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:25.841145 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000002 (ops 635-1258)
I20251026 03:12:25.841641 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:25.841753 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 91791 bytes on disk
I20251026 03:12:25.841907 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.841986 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014142
I20251026 03:12:25.844393 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.026s user 0.013s sys 0.000s Metrics: {"bytes_written":286830,"cfile_init":1,"dirs.queue_time_us":13,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":438,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20251026 03:12:25.844574 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014139
I20251026 03:12:25.853250 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.036s user 0.014s sys 0.000s Metrics: {"bytes_written":286830,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":520,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":127744}
I20251026 03:12:25.853399 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1075905 bytes of WAL
I20251026 03:12:25.853626 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:25.853660 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000002 (ops 672-1303)
I20251026 03:12:25.854393 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:25.857002 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 93708 bytes on disk
I20251026 03:12:25.857120 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.857218 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014139
I20251026 03:12:25.957571 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.113s user 0.075s sys 0.005s Metrics: {"bytes_written":1928685,"cfile_cache_hit":12,"cfile_cache_hit_bytes":645726,"cfile_cache_miss":94,"cfile_cache_miss_bytes":4240102,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":1136,"drs_written":1,"lbm_read_time_us":3124,"lbm_reads_lt_1ms":118,"lbm_write_time_us":4960,"lbm_writes_lt_1ms":99,"num_input_rowsets":2,"peak_mem_usage":1044478,"rows_written":68400}
I20251026 03:12:25.957873 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 692556 bytes on disk
I20251026 03:12:25.958096 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:25.978483 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.005568 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.024s user 0.012s sys 0.000s Metrics: {"bytes_written":276534,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":578,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20251026 03:12:26.005762 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 88390 bytes on disk
I20251026 03:12:26.005973 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.006287 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014080
I20251026 03:12:26.008605 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.167s user 0.066s sys 0.008s Metrics: {"bytes_written":1919937,"cfile_cache_hit":11,"cfile_cache_hit_bytes":610360,"cfile_cache_miss":95,"cfile_cache_miss_bytes":4250284,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":510,"drs_written":1,"lbm_read_time_us":1978,"lbm_reads_lt_1ms":115,"lbm_write_time_us":7633,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":98,"num_input_rowsets":2,"peak_mem_usage":1037493,"rows_written":68050,"spinlock_wait_cycles":384}
I20251026 03:12:26.008744 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.062812 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.052s user 0.014s sys 0.001s Metrics: {"bytes_written":331779,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":698,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1112,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11550}
I20251026 03:12:26.063022 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014067
I20251026 03:12:26.087538 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.230s user 0.072s sys 0.006s Metrics: {"bytes_written":1928685,"cfile_cache_hit":13,"cfile_cache_hit_bytes":808857,"cfile_cache_miss":93,"cfile_cache_miss_bytes":4076971,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":393,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":520,"drs_written":1,"lbm_read_time_us":5049,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":112,"lbm_write_time_us":12926,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":96,"mutex_wait_us":42,"num_input_rowsets":2,"peak_mem_usage":1044478,"rows_written":68400,"spinlock_wait_cycles":384}
I20251026 03:12:26.087761 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 692556 bytes on disk
I20251026 03:12:26.087885 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.087996 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.111392 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.105s user 0.078s sys 0.008s Metrics: {"bytes_written":2197419,"cfile_cache_hit":14,"cfile_cache_hit_bytes":867016,"cfile_cache_miss":104,"cfile_cache_miss_bytes":4724778,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1249,"dirs.run_cpu_time_us":409,"dirs.run_wall_time_us":4622,"drs_written":1,"lbm_read_time_us":2333,"lbm_reads_lt_1ms":124,"lbm_write_time_us":3895,"lbm_writes_lt_1ms":111,"num_input_rowsets":2,"peak_mem_usage":1191436,"rows_written":78000}
I20251026 03:12:26.111642 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 801666 bytes on disk
I20251026 03:12:26.111896 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.113071 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.025s user 0.015s sys 0.002s Metrics: {"bytes_written":429784,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":471,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":800,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":15000}
I20251026 03:12:26.113235 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014033
I20251026 03:12:26.162509 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.193006 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.130s user 0.080s sys 0.007s Metrics: {"bytes_written":2243236,"cfile_cache_hit":12,"cfile_cache_hit_bytes":698880,"cfile_cache_miss":108,"cfile_cache_miss_bytes":5006722,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":597,"drs_written":1,"lbm_read_time_us":2615,"lbm_reads_lt_1ms":136,"lbm_write_time_us":5329,"lbm_writes_lt_1ms":113,"num_input_rowsets":2,"peak_mem_usage":1212184,"rows_written":79600,"spinlock_wait_cycles":768}
I20251026 03:12:26.193176 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 819764 bytes on disk
I20251026 03:12:26.193293 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.193383 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.206792 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"bytes_written":277756,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":489,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":737,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20251026 03:12:26.206997 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 88829 bytes on disk
I20251026 03:12:26.207136 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.207279 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013761
I20251026 03:12:26.212400 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.050s user 0.012s sys 0.001s Metrics: {"bytes_written":286888,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":623,"drs_written":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4,"lbm_write_time_us":742,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":238336}
I20251026 03:12:26.212565 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1069528 bytes of WAL
I20251026 03:12:26.212657 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:26.212740 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000003 (ops 1157-1745)
I20251026 03:12:26.213109 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:26.213220 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.014001
I20251026 03:12:26.262213 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.149s user 0.077s sys 0.011s Metrics: {"bytes_written":2348711,"cfile_cache_hit":12,"cfile_cache_hit_bytes":752533,"cfile_cache_miss":112,"cfile_cache_miss_bytes":5225321,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":127,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":547,"drs_written":1,"lbm_read_time_us":1904,"lbm_reads_lt_1ms":136,"lbm_write_time_us":7449,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":117,"num_input_rowsets":2,"peak_mem_usage":1273777,"rows_written":83400}
I20251026 03:12:26.262388 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 862377 bytes on disk
I20251026 03:12:26.262533 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.299842 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.092s user 0.087s sys 0.003s Metrics: {"bytes_written":2512786,"cfile_cache_hit":11,"cfile_cache_hit_bytes":475124,"cfile_cache_miss":121,"cfile_cache_miss_bytes":5945832,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":111,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":670,"drs_written":1,"lbm_read_time_us":2609,"lbm_reads_lt_1ms":141,"lbm_write_time_us":3970,"lbm_writes_lt_1ms":125,"num_input_rowsets":2,"peak_mem_usage":1372530,"rows_written":89250}
I20251026 03:12:26.300035 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 928068 bytes on disk
I20251026 03:12:26.300175 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.304394 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.324734 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.020s user 0.013s sys 0.000s Metrics: {"bytes_written":286794,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":488,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":652,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000}
I20251026 03:12:26.325671 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 92057 bytes on disk
I20251026 03:12:26.325784 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.327092 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013946
I20251026 03:12:26.330883 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.117s user 0.086s sys 0.008s Metrics: {"bytes_written":2476352,"cfile_cache_hit":12,"cfile_cache_hit_bytes":811587,"cfile_cache_miss":118,"cfile_cache_miss_bytes":5514863,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":64,"dirs.run_wall_time_us":753,"drs_written":1,"lbm_read_time_us":2651,"lbm_reads_lt_1ms":142,"lbm_write_time_us":9669,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":122,"num_input_rowsets":2,"peak_mem_usage":1353526,"rows_written":88000}
I20251026 03:12:26.331264 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 914354 bytes on disk
I20251026 03:12:26.331418 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.382052 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.401568 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.411113 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.011s sys 0.001s Metrics: {"bytes_written":289740,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":644,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10100}
I20251026 03:12:26.411263 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 93088 bytes on disk
I20251026 03:12:26.411473 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.411616 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013923
I20251026 03:12:26.416636 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.015s user 0.005s sys 0.004s Metrics: {"bytes_written":281638,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":536,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":580,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800,"spinlock_wait_cycles":172416}
I20251026 03:12:26.416817 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1049237 bytes of WAL
I20251026 03:12:26.416935 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:26.417022 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000003 (ops 1259-1878)
I20251026 03:12:26.417404 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:26.417600 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013912
I20251026 03:12:26.431231 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.104s user 0.093s sys 0.003s Metrics: {"bytes_written":2627878,"cfile_cache_hit":10,"cfile_cache_hit_bytes":451787,"cfile_cache_miss":128,"cfile_cache_miss_bytes":6273415,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":447,"drs_written":1,"lbm_read_time_us":2539,"lbm_reads_lt_1ms":148,"lbm_write_time_us":4514,"lbm_writes_lt_1ms":129,"mutex_wait_us":3,"num_input_rowsets":2,"peak_mem_usage":1441268,"rows_written":93400}
I20251026 03:12:26.431356 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 974724 bytes on disk
I20251026 03:12:26.431448 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.482316 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.537237 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.124s user 0.096s sys 0.008s Metrics: {"bytes_written":2759905,"cfile_cache_hit":12,"cfile_cache_hit_bytes":810513,"cfile_cache_miss":130,"cfile_cache_miss_bytes":6257163,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":358,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":704,"drs_written":1,"lbm_read_time_us":2795,"lbm_reads_lt_1ms":150,"lbm_write_time_us":4531,"lbm_writes_lt_1ms":135,"num_input_rowsets":2,"peak_mem_usage":1517536,"rows_written":98100,"spinlock_wait_cycles":2048}
I20251026 03:12:26.537490 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1027371 bytes on disk
I20251026 03:12:26.537657 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.538964 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.056s user 0.013s sys 0.000s Metrics: {"bytes_written":272525,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":616,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2478,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9450}
I20251026 03:12:26.539105 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1050675 bytes of WAL
I20251026 03:12:26.539163 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:26.539180 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000003 (ops 1304-1922)
I20251026 03:12:26.541292 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:26.541384 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 87043 bytes on disk
I20251026 03:12:26.541913 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.542011 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013739
I20251026 03:12:26.561667 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.144s user 0.094s sys 0.008s Metrics: {"bytes_written":2783962,"cfile_cache_hit":10,"cfile_cache_hit_bytes":447998,"cfile_cache_miss":134,"cfile_cache_miss_bytes":6689918,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":137,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":727,"drs_written":1,"lbm_read_time_us":4072,"lbm_reads_lt_1ms":158,"lbm_write_time_us":5157,"lbm_writes_lt_1ms":135,"num_input_rowsets":2,"peak_mem_usage":1531625,"rows_written":99050}
I20251026 03:12:26.561846 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1037249 bytes on disk
I20251026 03:12:26.561970 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.578611 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.594341 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.595959 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.017s user 0.010s sys 0.001s Metrics: {"bytes_written":277749,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":670,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":607,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20251026 03:12:26.596074 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013662
I20251026 03:12:26.620607 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.026s user 0.007s sys 0.004s Metrics: {"bytes_written":271077,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":568,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20251026 03:12:26.621088 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 86462 bytes on disk
I20251026 03:12:26.621255 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.621567 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013655
I20251026 03:12:26.662268 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.120s user 0.104s sys 0.004s Metrics: {"bytes_written":2891920,"cfile_cache_hit":12,"cfile_cache_hit_bytes":796516,"cfile_cache_miss":136,"cfile_cache_miss_bytes":6623726,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":129,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":566,"drs_written":1,"lbm_read_time_us":2543,"lbm_reads_lt_1ms":156,"lbm_write_time_us":4819,"lbm_writes_lt_1ms":140,"num_input_rowsets":2,"peak_mem_usage":1593718,"rows_written":102850,"spinlock_wait_cycles":896}
I20251026 03:12:26.662401 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1079670 bytes on disk
I20251026 03:12:26.662592 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.683100 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.708700 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.026s user 0.013s sys 0.001s Metrics: {"bytes_written":284434,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":725,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":716,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20251026 03:12:26.708956 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013643
I20251026 03:12:26.769851 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.174s user 0.116s sys 0.001s Metrics: {"bytes_written":3030921,"cfile_cache_hit":13,"cfile_cache_hit_bytes":829049,"cfile_cache_miss":141,"cfile_cache_miss_bytes":6949835,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":641,"drs_written":1,"lbm_read_time_us":3976,"lbm_reads_lt_1ms":165,"lbm_write_time_us":7779,"lbm_writes_lt_1ms":146,"num_input_rowsets":2,"peak_mem_usage":1673136,"rows_written":107750}
I20251026 03:12:26.770180 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.787847 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.012s sys 0.000s Metrics: {"bytes_written":273760,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":561,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1050,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20251026 03:12:26.788070 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1067984 bytes of WAL
I20251026 03:12:26.788203 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:26.788266 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000004 (ops 1746-2333)
I20251026 03:12:26.788755 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:26.789629 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1221777 bytes on disk
I20251026 03:12:26.789875 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":73,"lbm_reads_lt_1ms":8}
I20251026 03:12:26.790025 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013761
I20251026 03:12:26.851296 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.142s user 0.121s sys 0.000s Metrics: {"bytes_written":3165674,"cfile_cache_hit":10,"cfile_cache_hit_bytes":450070,"cfile_cache_miss":148,"cfile_cache_miss_bytes":7690800,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":723,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":614,"drs_written":1,"lbm_read_time_us":3530,"lbm_reads_lt_1ms":172,"lbm_write_time_us":6281,"lbm_writes_lt_1ms":151,"num_input_rowsets":2,"peak_mem_usage":1754134,"rows_written":112750,"spinlock_wait_cycles":384}
I20251026 03:12:26.851495 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1189009 bytes on disk
I20251026 03:12:26.851668 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.882315 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.905885 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.284s user 0.116s sys 0.004s Metrics: {"bytes_written":3048929,"cfile_cache_hit":12,"cfile_cache_hit_bytes":647039,"cfile_cache_miss":142,"cfile_cache_miss_bytes":7181643,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":561,"drs_written":1,"lbm_read_time_us":4644,"lbm_reads_lt_1ms":162,"lbm_write_time_us":14992,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":143,"num_input_rowsets":2,"peak_mem_usage":1684766,"rows_written":108450,"spinlock_wait_cycles":768}
I20251026 03:12:26.906138 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1142139 bytes on disk
I20251026 03:12:26.906276 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.906420 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:26.921212 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.037s user 0.014s sys 0.000s Metrics: {"bytes_written":279180,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":67,"dirs.run_wall_time_us":893,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1282,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20251026 03:12:26.921826 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013716
I20251026 03:12:26.933466 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.014s sys 0.004s Metrics: {"bytes_written":428444,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":498,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14950}
I20251026 03:12:26.933650 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013707
I20251026 03:12:26.939747 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.150s user 0.126s sys 0.003s Metrics: {"bytes_written":3292241,"cfile_cache_hit":13,"cfile_cache_hit_bytes":824377,"cfile_cache_miss":151,"cfile_cache_miss_bytes":7650307,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":428,"drs_written":1,"lbm_read_time_us":3296,"lbm_reads_lt_1ms":171,"lbm_write_time_us":6965,"lbm_writes_lt_1ms":156,"num_input_rowsets":2,"peak_mem_usage":1827651,"rows_written":117250}
I20251026 03:12:26.939926 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1238706 bytes on disk
I20251026 03:12:26.940081 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:26.990867 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.032116 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.041s user 0.013s sys 0.000s Metrics: {"bytes_written":275133,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":674,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":594,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550}
I20251026 03:12:27.032343 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013443
I20251026 03:12:27.066656 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.133s user 0.128s sys 0.000s Metrics: {"bytes_written":3467050,"cfile_cache_hit":10,"cfile_cache_hit_bytes":525858,"cfile_cache_miss":160,"cfile_cache_miss_bytes":8394666,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":515,"drs_written":1,"lbm_read_time_us":3011,"lbm_reads_lt_1ms":184,"lbm_write_time_us":5167,"lbm_writes_lt_1ms":164,"num_input_rowsets":2,"peak_mem_usage":1920477,"rows_written":123400}
I20251026 03:12:27.066917 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1309996 bytes on disk
I20251026 03:12:27.067050 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.069455 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.147s user 0.110s sys 0.019s Metrics: {"bytes_written":3438220,"cfile_cache_hit":11,"cfile_cache_hit_bytes":621470,"cfile_cache_miss":159,"cfile_cache_miss_bytes":8233166,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":64,"dirs.run_wall_time_us":512,"drs_written":1,"lbm_read_time_us":3508,"lbm_reads_lt_1ms":183,"lbm_write_time_us":5192,"lbm_writes_lt_1ms":162,"num_input_rowsets":2,"peak_mem_usage":1911949,"rows_written":122450}
I20251026 03:12:27.069597 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1299214 bytes on disk
I20251026 03:12:27.070350 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.110877 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.127432 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.016s user 0.011s sys 0.000s Metrics: {"bytes_written":276275,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":799,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":545,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20251026 03:12:27.127650 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1052750 bytes of WAL
I20251026 03:12:27.127776 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:27.127789 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.134039 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000004 (ops 1923-2548)
I20251026 03:12:27.134523 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:27.145437 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013447
I20251026 03:12:27.141032 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":271049,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":502,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9400}
I20251026 03:12:27.145639 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1052603 bytes of WAL
I20251026 03:12:27.145908 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:27.145979 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000004 (ops 1879-2506)
I20251026 03:12:27.146404 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:27.150336 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 86556 bytes on disk
I20251026 03:12:27.151856 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.151990 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013438
I20251026 03:12:27.200309 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.168s user 0.121s sys 0.015s Metrics: {"bytes_written":3560572,"cfile_cache_hit":10,"cfile_cache_hit_bytes":443487,"cfile_cache_miss":164,"cfile_cache_miss_bytes":8725151,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":605,"drs_written":1,"lbm_read_time_us":4257,"lbm_reads_lt_1ms":188,"lbm_write_time_us":6910,"lbm_writes_lt_1ms":168,"num_input_rowsets":2,"peak_mem_usage":1981127,"rows_written":126800,"spinlock_wait_cycles":384}
I20251026 03:12:27.200515 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1349410 bytes on disk
I20251026 03:12:27.200717 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.210963 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.228730 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":276422,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":545,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":684,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9600}
I20251026 03:12:27.230377 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 88429 bytes on disk
I20251026 03:12:27.230713 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.230849 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013400
I20251026 03:12:27.368436 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.222s user 0.130s sys 0.000s Metrics: {"bytes_written":3706469,"cfile_cache_hit":15,"cfile_cache_hit_bytes":981330,"cfile_cache_miss":165,"cfile_cache_miss_bytes":8578784,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":645,"drs_written":1,"lbm_read_time_us":13553,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":188,"lbm_write_time_us":7095,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":175,"num_input_rowsets":2,"peak_mem_usage":2068039,"rows_written":132050,"spinlock_wait_cycles":384}
I20251026 03:12:27.368710 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.382890 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":283162,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":430,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":650,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9850}
I20251026 03:12:27.383111 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1501105 bytes on disk
I20251026 03:12:27.383296 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":30,"lbm_reads_lt_1ms":8}
I20251026 03:12:27.383525 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013099
I20251026 03:12:27.405251 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.174s user 0.127s sys 0.012s Metrics: {"bytes_written":3826952,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1118198,"cfile_cache_miss":172,"cfile_cache_miss_bytes":8762602,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":542,"drs_written":1,"lbm_read_time_us":4837,"lbm_reads_lt_1ms":192,"lbm_write_time_us":6974,"lbm_writes_lt_1ms":180,"num_input_rowsets":2,"peak_mem_usage":2140372,"rows_written":136400}
I20251026 03:12:27.405421 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1460359 bytes on disk
I20251026 03:12:27.408875 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.437748 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.286s user 0.132s sys 0.010s Metrics: {"bytes_written":3725470,"cfile_cache_hit":12,"cfile_cache_hit_bytes":644183,"cfile_cache_miss":170,"cfile_cache_miss_bytes":8972731,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":432,"drs_written":1,"lbm_read_time_us":4443,"lbm_reads_lt_1ms":190,"lbm_write_time_us":7607,"lbm_writes_lt_1ms":176,"num_input_rowsets":2,"peak_mem_usage":2082073,"rows_written":132800,"spinlock_wait_cycles":920704}
I20251026 03:12:27.437929 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1418875 bytes on disk
I20251026 03:12:27.438041 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.438140 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.460232 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.461752 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.024s user 0.013s sys 0.003s Metrics: {"bytes_written":351875,"cfile_init":1,"dirs.queue_time_us":25,"dirs.run_cpu_time_us":68,"dirs.run_wall_time_us":564,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1084,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12150}
I20251026 03:12:27.461948 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013416
I20251026 03:12:27.505261 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.044s user 0.015s sys 0.000s Metrics: {"bytes_written":281462,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":745,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1643,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20251026 03:12:27.505535 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013401
I20251026 03:12:27.556063 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.172s user 0.139s sys 0.016s Metrics: {"bytes_written":3977131,"cfile_cache_hit":11,"cfile_cache_hit_bytes":622856,"cfile_cache_miss":185,"cfile_cache_miss_bytes":9665004,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":91,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":654,"drs_written":1,"lbm_read_time_us":3889,"lbm_reads_lt_1ms":205,"lbm_write_time_us":6938,"lbm_writes_lt_1ms":187,"num_input_rowsets":2,"peak_mem_usage":2224534,"rows_written":141900}
I20251026 03:12:27.556355 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1523289 bytes on disk
I20251026 03:12:27.556516 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.607409 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.622223 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":279224,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":544,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":709,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9700}
I20251026 03:12:27.622474 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013241
I20251026 03:12:27.678040 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.216s user 0.147s sys 0.016s Metrics: {"bytes_written":4063927,"cfile_cache_hit":13,"cfile_cache_hit_bytes":877815,"cfile_cache_miss":185,"cfile_cache_miss_bytes":9630253,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":515,"drs_written":1,"lbm_read_time_us":10255,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":208,"lbm_write_time_us":9353,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":189,"num_input_rowsets":2,"peak_mem_usage":2275097,"rows_written":144950,"spinlock_wait_cycles":1280}
I20251026 03:12:27.678897 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1557713 bytes on disk
I20251026 03:12:27.679051 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.699414 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.727451 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.010s sys 0.003s Metrics: {"bytes_written":272576,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":536,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":703,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20251026 03:12:27.728261 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013133
I20251026 03:12:27.745780 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.240s user 0.147s sys 0.012s Metrics: {"bytes_written":4099372,"cfile_cache_hit":12,"cfile_cache_hit_bytes":652485,"cfile_cache_miss":188,"cfile_cache_miss_bytes":9955401,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":780,"drs_written":1,"lbm_read_time_us":7030,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":211,"lbm_write_time_us":9237,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":191,"num_input_rowsets":2,"peak_mem_usage":2299407,"rows_written":146200,"spinlock_wait_cycles":7936}
I20251026 03:12:27.745935 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1572432 bytes on disk
I20251026 03:12:27.746049 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.756235 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.783810 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.161s user 0.142s sys 0.016s Metrics: {"bytes_written":4250747,"cfile_cache_hit":10,"cfile_cache_hit_bytes":446708,"cfile_cache_miss":198,"cfile_cache_miss_bytes":10556098,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":64,"dirs.run_wall_time_us":666,"drs_written":1,"lbm_read_time_us":4364,"lbm_reads_lt_1ms":222,"lbm_write_time_us":6888,"lbm_writes_lt_1ms":198,"num_input_rowsets":2,"peak_mem_usage":2387254,"rows_written":151600}
I20251026 03:12:27.784080 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1633144 bytes on disk
I20251026 03:12:27.784257 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.793303 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.037s user 0.012s sys 0.000s Metrics: {"bytes_written":281597,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":593,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1718,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800}
I20251026 03:12:27.793744 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 90290 bytes on disk
I20251026 03:12:27.793957 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.794096 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1104397 bytes of WAL
I20251026 03:12:27.794180 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:27.794315 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000005 (ops 2334-2941)
I20251026 03:12:27.794734 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:27.794857 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013117
I20251026 03:12:27.875385 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.889045 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.014s user 0.010s sys 0.000s Metrics: {"bytes_written":277825,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":627,"drs_written":1,"lbm_read_time_us":24,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20251026 03:12:27.889233 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1057818 bytes of WAL
I20251026 03:12:27.889317 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:27.889357 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000005 (ops 2549-3152)
I20251026 03:12:27.889981 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:27.890107 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012898
I20251026 03:12:27.937275 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.209s user 0.152s sys 0.012s Metrics: {"bytes_written":4329237,"cfile_cache_hit":13,"cfile_cache_hit_bytes":817093,"cfile_cache_miss":197,"cfile_cache_miss_bytes":10393615,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":125,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":490,"drs_written":1,"lbm_read_time_us":6103,"lbm_reads_lt_1ms":221,"lbm_write_time_us":7332,"lbm_writes_lt_1ms":202,"num_input_rowsets":2,"peak_mem_usage":2429698,"rows_written":154400,"spinlock_wait_cycles":768}
I20251026 03:12:27.937618 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1664754 bytes on disk
I20251026 03:12:27.937750 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:27.969439 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:27.992622 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.023s user 0.012s sys 0.000s Metrics: {"bytes_written":273726,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":688,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20251026 03:12:27.992789 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1085285 bytes of WAL
I20251026 03:12:27.993036 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:27.993091 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000005 (ops 2507-3133)
I20251026 03:12:27.993934 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:27.994050 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012869
I20251026 03:12:28.001869 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.206s user 0.153s sys 0.007s Metrics: {"bytes_written":4372739,"cfile_cache_hit":12,"cfile_cache_hit_bytes":651588,"cfile_cache_miss":200,"cfile_cache_miss_bytes":10676292,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":585,"drs_written":1,"lbm_read_time_us":5596,"lbm_reads_lt_1ms":220,"lbm_write_time_us":7290,"lbm_writes_lt_1ms":204,"num_input_rowsets":2,"peak_mem_usage":2458647,"rows_written":156000,"spinlock_wait_cycles":848384}
I20251026 03:12:28.002072 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1683083 bytes on disk
I20251026 03:12:28.002214 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.032732 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.053902 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.021s user 0.005s sys 0.008s Metrics: {"bytes_written":280345,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":645,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":961,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750}
I20251026 03:12:28.054152 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013122
I20251026 03:12:28.094697 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.204s user 0.148s sys 0.015s Metrics: {"bytes_written":4520105,"cfile_cache_hit":12,"cfile_cache_hit_bytes":790253,"cfile_cache_miss":206,"cfile_cache_miss_bytes":10928763,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":73,"dirs.run_wall_time_us":655,"drs_written":1,"lbm_read_time_us":4882,"lbm_reads_lt_1ms":230,"lbm_write_time_us":7804,"lbm_writes_lt_1ms":209,"num_input_rowsets":2,"peak_mem_usage":2542459,"rows_written":161250}
I20251026 03:12:28.094924 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1742820 bytes on disk
I20251026 03:12:28.095218 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.145913 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.169287 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.023s user 0.011s sys 0.003s Metrics: {"bytes_written":293569,"cfile_init":1,"dirs.queue_time_us":2800,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":769,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":622,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20251026 03:12:28.169589 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013111
I20251026 03:12:28.238739 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.184s user 0.154s sys 0.016s Metrics: {"bytes_written":4643794,"cfile_cache_hit":13,"cfile_cache_hit_bytes":822008,"cfile_cache_miss":211,"cfile_cache_miss_bytes":11223394,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":544,"drs_written":1,"lbm_read_time_us":4930,"lbm_reads_lt_1ms":235,"lbm_write_time_us":8416,"lbm_writes_lt_1ms":214,"num_input_rowsets":2,"peak_mem_usage":2615472,"rows_written":165750,"spinlock_wait_cycles":896}
I20251026 03:12:28.239022 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1795029 bytes on disk
I20251026 03:12:28.239195 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.239627 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.257648 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":293447,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":455,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1384,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20251026 03:12:28.257900 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1051415 bytes of WAL
I20251026 03:12:28.258548 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:28.258615 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000006 (ops 2942-3520)
I20251026 03:12:28.258946 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:28.259052 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 94453 bytes on disk
I20251026 03:12:28.259172 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.259263 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013140
I20251026 03:12:28.311874 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.318s user 0.142s sys 0.023s Metrics: {"bytes_written":4591925,"cfile_cache_hit":13,"cfile_cache_hit_bytes":816828,"cfile_cache_miss":209,"cfile_cache_miss_bytes":11095656,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":70,"dirs.run_wall_time_us":447,"drs_written":1,"lbm_read_time_us":10036,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":232,"lbm_write_time_us":12487,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":211,"num_input_rowsets":2,"peak_mem_usage":2587313,"rows_written":163900,"spinlock_wait_cycles":384}
I20251026 03:12:28.312150 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.339619 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.017s sys 0.000s Metrics: {"bytes_written":423559,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":623,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1040,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14750}
I20251026 03:12:28.339875 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013114
I20251026 03:12:28.435894 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.266s user 0.156s sys 0.026s Metrics: {"bytes_written":4803800,"cfile_cache_hit":11,"cfile_cache_hit_bytes":489500,"cfile_cache_miss":217,"cfile_cache_miss_bytes":11978728,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":785,"drs_written":1,"lbm_read_time_us":6015,"lbm_reads_lt_1ms":241,"lbm_write_time_us":10471,"lbm_writes_lt_1ms":221,"num_input_rowsets":2,"peak_mem_usage":2706755,"rows_written":171500,"spinlock_wait_cycles":896}
I20251026 03:12:28.436203 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.457437 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":326515,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":576,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":715,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11350}
I20251026 03:12:28.457674 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.013068
I20251026 03:12:28.467761 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.208s user 0.175s sys 0.019s Metrics: {"bytes_written":4927380,"cfile_cache_hit":14,"cfile_cache_hit_bytes":841819,"cfile_cache_miss":220,"cfile_cache_miss_bytes":11961501,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":651,"drs_written":1,"lbm_read_time_us":4980,"lbm_reads_lt_1ms":240,"lbm_write_time_us":9538,"lbm_writes_lt_1ms":226,"num_input_rowsets":2,"peak_mem_usage":2779872,"rows_written":176000,"spinlock_wait_cycles":1408}
I20251026 03:12:28.468364 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1913704 bytes on disk
I20251026 03:12:28.468658 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.509869 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.556804 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.046s user 0.013s sys 0.000s Metrics: {"bytes_written":286791,"cfile_init":1,"dirs.queue_time_us":530,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":530,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":915,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":148736}
I20251026 03:12:28.557565 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 92196 bytes on disk
I20251026 03:12:28.557693 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.558235 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012939
I20251026 03:12:28.636876 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.297s user 0.166s sys 0.021s Metrics: {"bytes_written":5003789,"cfile_cache_hit":13,"cfile_cache_hit_bytes":914207,"cfile_cache_miss":225,"cfile_cache_miss_bytes":12073975,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":1737,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":730,"drs_written":1,"lbm_read_time_us":9692,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":252,"lbm_write_time_us":9829,"lbm_writes_lt_1ms":229,"num_input_rowsets":2,"peak_mem_usage":2820079,"rows_written":178650,"spinlock_wait_cycles":9840512}
I20251026 03:12:28.637411 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1944398 bytes on disk
I20251026 03:12:28.638588 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.639038 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.658994 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.020s user 0.015s sys 0.000s Metrics: {"bytes_written":364027,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":671,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12600}
I20251026 03:12:28.659214 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1051371 bytes of WAL
I20251026 03:12:28.659346 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:28.659440 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000006 (ops 3134-3751)
I20251026 03:12:28.659862 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:28.660005 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012945
I20251026 03:12:28.690706 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.233s user 0.167s sys 0.016s Metrics: {"bytes_written":5117012,"cfile_cache_hit":13,"cfile_cache_hit_bytes":790287,"cfile_cache_miss":229,"cfile_cache_miss_bytes":12513145,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":485,"drs_written":1,"lbm_read_time_us":5117,"lbm_reads_lt_1ms":257,"lbm_write_time_us":9865,"lbm_writes_lt_1ms":234,"num_input_rowsets":2,"peak_mem_usage":2891728,"rows_written":182850,"spinlock_wait_cycles":5378944}
I20251026 03:12:28.690954 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 1992491 bytes on disk
I20251026 03:12:28.691105 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.691221 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.705839 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":293517,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":434,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10250}
I20251026 03:12:28.706010 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1063417 bytes of WAL
I20251026 03:12:28.706113 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:28.706687 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000006 (ops 3153-3777)
I20251026 03:12:28.707084 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:28.707194 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 94583 bytes on disk
I20251026 03:12:28.707309 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.707423 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012793
I20251026 03:12:28.886096 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.226s user 0.197s sys 0.000s Metrics: {"bytes_written":5351205,"cfile_cache_hit":13,"cfile_cache_hit_bytes":789033,"cfile_cache_miss":237,"cfile_cache_miss_bytes":13137171,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":797,"drs_written":1,"lbm_read_time_us":5640,"lbm_reads_lt_1ms":261,"lbm_write_time_us":9214,"lbm_writes_lt_1ms":243,"num_input_rowsets":2,"peak_mem_usage":3026969,"rows_written":191250}
I20251026 03:12:28.886390 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2088600 bytes on disk
I20251026 03:12:28.886557 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.886687 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.905025 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.013s sys 0.000s Metrics: {"bytes_written":284289,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":553,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":577,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9900}
I20251026 03:12:28.905246 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 91323 bytes on disk
I20251026 03:12:28.905382 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.905519 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012723
I20251026 03:12:28.924021 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.366s user 0.188s sys 0.013s Metrics: {"bytes_written":5204519,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1058221,"cfile_cache_miss":231,"cfile_cache_miss_bytes":12482353,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":125,"dirs.run_cpu_time_us":396,"dirs.run_wall_time_us":2954,"drs_written":1,"lbm_read_time_us":22763,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":249,"lbm_write_time_us":13424,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":236,"num_input_rowsets":2,"peak_mem_usage":2945912,"rows_written":186000,"spinlock_wait_cycles":896}
I20251026 03:12:28.925127 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2027751 bytes on disk
I20251026 03:12:28.925313 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:28.925453 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:28.957206 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.032s user 0.021s sys 0.000s Metrics: {"bytes_written":482961,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":573,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1286,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16950}
I20251026 03:12:28.957456 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012873
I20251026 03:12:29.153573 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.446s user 0.188s sys 0.015s Metrics: {"bytes_written":5402895,"cfile_cache_hit":13,"cfile_cache_hit_bytes":794139,"cfile_cache_miss":241,"cfile_cache_miss_bytes":13268565,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":759,"drs_written":1,"lbm_read_time_us":9402,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":260,"lbm_write_time_us":28241,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":242,"num_input_rowsets":2,"peak_mem_usage":3057729,"rows_written":193100}
I20251026 03:12:29.153899 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.181932 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.021s sys 0.000s Metrics: {"bytes_written":521293,"cfile_init":1,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":603,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1229,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18300}
I20251026 03:12:29.182161 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012778
I20251026 03:12:29.194445 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.289s user 0.206s sys 0.008s Metrics: {"bytes_written":5626461,"cfile_cache_hit":11,"cfile_cache_hit_bytes":481934,"cfile_cache_miss":251,"cfile_cache_miss_bytes":14175198,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":3771,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":804,"drs_written":1,"lbm_read_time_us":7659,"lbm_reads_lt_1ms":271,"lbm_write_time_us":10487,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":256,"num_input_rowsets":2,"peak_mem_usage":3190535,"rows_written":201150}
I20251026 03:12:29.194690 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2201363 bytes on disk
I20251026 03:12:29.194993 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.195147 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.224128 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.029s user 0.012s sys 0.001s Metrics: {"bytes_written":335847,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":592,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11700}
I20251026 03:12:29.224354 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012760
I20251026 03:12:29.272197 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.315s user 0.208s sys 0.014s Metrics: {"bytes_written":5676711,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1256225,"cfile_cache_miss":251,"cfile_cache_miss_bytes":13514007,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":538,"drs_written":1,"lbm_read_time_us":21218,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":272,"lbm_write_time_us":14659,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":258,"num_input_rowsets":2,"peak_mem_usage":3212726,"rows_written":202950,"spinlock_wait_cycles":2600832}
I20251026 03:12:29.272553 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2222554 bytes on disk
I20251026 03:12:29.273217 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.273377 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.293226 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.020s user 0.010s sys 0.007s Metrics: {"bytes_written":378181,"cfile_init":1,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":616,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2052,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13150}
I20251026 03:12:29.293452 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1093334 bytes of WAL
I20251026 03:12:29.293561 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:29.293612 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000007 (ops 3521-4122)
I20251026 03:12:29.294055 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:29.294203 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 122519 bytes on disk
I20251026 03:12:29.294348 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.294471 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012603
I20251026 03:12:29.471568 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.289s user 0.199s sys 0.023s Metrics: {"bytes_written":5910376,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1070733,"cfile_cache_miss":260,"cfile_cache_miss_bytes":14325845,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":754,"drs_written":1,"lbm_read_time_us":8137,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":287,"lbm_write_time_us":13921,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":267,"num_input_rowsets":2,"peak_mem_usage":3350326,"rows_written":211400}
I20251026 03:12:29.473407 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2320889 bytes on disk
I20251026 03:12:29.473582 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.473796 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.501689 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.005s sys 0.008s Metrics: {"bytes_written":331789,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":1655,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":708,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11550}
I20251026 03:12:29.501974 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1054312 bytes of WAL
I20251026 03:12:29.502501 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:29.502589 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000007 (ops 3778-4396)
I20251026 03:12:29.503250 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:29.503369 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 107275 bytes on disk
I20251026 03:12:29.503556 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.503786 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012416
I20251026 03:12:29.509093 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.285s user 0.200s sys 0.028s Metrics: {"bytes_written":5950984,"cfile_cache_hit":13,"cfile_cache_hit_bytes":859145,"cfile_cache_miss":267,"cfile_cache_miss_bytes":14653885,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":497,"dirs.run_wall_time_us":3377,"drs_written":1,"lbm_read_time_us":7743,"lbm_reads_lt_1ms":291,"lbm_write_time_us":10965,"lbm_writes_lt_1ms":271,"mutex_wait_us":45,"num_input_rowsets":2,"peak_mem_usage":3382143,"rows_written":212850,"spinlock_wait_cycles":1536}
I20251026 03:12:29.509305 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2337346 bytes on disk
I20251026 03:12:29.512207 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":350,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.512372 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.551061 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.037s user 0.014s sys 0.000s Metrics: {"bytes_written":334492,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":624,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2258,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":11650}
I20251026 03:12:29.551290 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1084170 bytes of WAL
I20251026 03:12:29.551401 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:29.551456 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000007 (ops 3752-4388)
I20251026 03:12:29.551855 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:29.551963 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012399
I20251026 03:12:29.623586 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.329s user 0.199s sys 0.031s Metrics: {"bytes_written":6042914,"cfile_cache_hit":11,"cfile_cache_hit_bytes":554066,"cfile_cache_miss":273,"cfile_cache_miss_bytes":15197992,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":168,"dirs.run_cpu_time_us":452,"dirs.run_wall_time_us":3577,"drs_written":1,"lbm_read_time_us":10012,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":292,"lbm_write_time_us":20073,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":269,"num_input_rowsets":2,"peak_mem_usage":3431850,"rows_written":216100}
I20251026 03:12:29.623831 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2374712 bytes on disk
I20251026 03:12:29.624787 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.624953 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.667641 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.043s user 0.013s sys 0.004s Metrics: {"bytes_written":384853,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":684,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":952,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13400}
I20251026 03:12:29.667883 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012421
I20251026 03:12:29.799072 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.295s user 0.209s sys 0.033s Metrics: {"bytes_written":6233308,"cfile_cache_hit":11,"cfile_cache_hit_bytes":518901,"cfile_cache_miss":281,"cfile_cache_miss_bytes":15740931,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":276,"dirs.run_cpu_time_us":125,"dirs.run_wall_time_us":980,"drs_written":1,"lbm_read_time_us":8148,"lbm_reads_lt_1ms":301,"lbm_write_time_us":13431,"lbm_writes_lt_1ms":283,"num_input_rowsets":2,"peak_mem_usage":3546408,"rows_written":222950,"spinlock_wait_cycles":1664}
I20251026 03:12:29.799355 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2454201 bytes on disk
I20251026 03:12:29.799530 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.799695 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.813458 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":363968,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":687,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12600}
I20251026 03:12:29.813669 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 117388 bytes on disk
I20251026 03:12:29.813848 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:29.813961 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012415
I20251026 03:12:29.881635 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.330s user 0.218s sys 0.010s Metrics: {"bytes_written":6275776,"cfile_cache_hit":11,"cfile_cache_hit_bytes":521193,"cfile_cache_miss":283,"cfile_cache_miss_bytes":15851653,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":28,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":622,"drs_written":1,"lbm_read_time_us":8020,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":306,"lbm_write_time_us":15662,"lbm_writes_lt_1ms":285,"num_input_rowsets":2,"peak_mem_usage":3569842,"rows_written":224500,"spinlock_wait_cycles":3200}
I20251026 03:12:29.882588 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:29.922057 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.039s user 0.012s sys 0.004s Metrics: {"bytes_written":399542,"cfile_init":1,"dirs.queue_time_us":356,"dirs.run_cpu_time_us":75,"dirs.run_wall_time_us":904,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1457,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":13800}
I20251026 03:12:29.922255 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2600868 bytes on disk
I20251026 03:12:29.922539 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":39,"lbm_reads_lt_1ms":8}
I20251026 03:12:29.922689 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012436
I20251026 03:12:30.002485 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.334s user 0.218s sys 0.021s Metrics: {"bytes_written":6415764,"cfile_cache_hit":12,"cfile_cache_hit_bytes":727903,"cfile_cache_miss":286,"cfile_cache_miss_bytes":16013787,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":683,"drs_written":1,"lbm_read_time_us":8163,"lbm_reads_lt_1ms":310,"lbm_write_time_us":13037,"lbm_writes_lt_1ms":290,"num_input_rowsets":2,"peak_mem_usage":3652639,"rows_written":229500}
I20251026 03:12:30.005211 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2529375 bytes on disk
I20251026 03:12:30.005421 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.005589 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.032843 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.017s sys 0.001s Metrics: {"bytes_written":379645,"cfile_init":1,"dirs.queue_time_us":27,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":635,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13200}
I20251026 03:12:30.033383 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1051637 bytes of WAL
I20251026 03:12:30.033568 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:30.033643 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000008 (ops 4123-4701)
I20251026 03:12:30.034206 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:30.034335 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 124290 bytes on disk
I20251026 03:12:30.034788 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.034933 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012390
I20251026 03:12:30.134562 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.321s user 0.222s sys 0.022s Metrics: {"bytes_written":6582311,"cfile_cache_hit":11,"cfile_cache_hit_bytes":670745,"cfile_cache_miss":295,"cfile_cache_miss_bytes":16520251,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":598,"drs_written":1,"lbm_read_time_us":9598,"lbm_reads_lt_1ms":315,"lbm_write_time_us":17527,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":295,"num_input_rowsets":2,"peak_mem_usage":3750092,"rows_written":235550,"spinlock_wait_cycles":384}
I20251026 03:12:30.135990 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2598831 bytes on disk
I20251026 03:12:30.137348 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.137606 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.162035 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.024s user 0.014s sys 0.001s Metrics: {"bytes_written":316008,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":547,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":956,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":10950}
I20251026 03:12:30.162340 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012411
I20251026 03:12:30.190860 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.268s user 0.229s sys 0.020s Metrics: {"bytes_written":6658009,"cfile_cache_hit":12,"cfile_cache_hit_bytes":861276,"cfile_cache_miss":298,"cfile_cache_miss_bytes":16527892,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1086,"dirs.run_cpu_time_us":57,"dirs.run_wall_time_us":13829,"drs_written":1,"lbm_read_time_us":7057,"lbm_reads_lt_1ms":318,"lbm_write_time_us":12228,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":299,"num_input_rowsets":2,"peak_mem_usage":3795224,"rows_written":238300,"spinlock_wait_cycles":384}
I20251026 03:12:30.191071 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2629986 bytes on disk
I20251026 03:12:30.191323 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.191414 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.213644 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.009s sys 0.004s Metrics: {"bytes_written":277791,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":683,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":515,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20251026 03:12:30.213900 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012070
I20251026 03:12:30.438120 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.275s user 0.226s sys 0.022s Metrics: {"bytes_written":6888733,"cfile_cache_hit":11,"cfile_cache_hit_bytes":638729,"cfile_cache_miss":309,"cfile_cache_miss_bytes":17365455,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":635,"drs_written":1,"lbm_read_time_us":7327,"lbm_reads_lt_1ms":333,"lbm_write_time_us":12387,"lbm_writes_lt_1ms":310,"num_input_rowsets":2,"peak_mem_usage":3928021,"rows_written":246500}
I20251026 03:12:30.438716 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2725290 bytes on disk
I20251026 03:12:30.438933 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.439386 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.442658 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.408s user 0.230s sys 0.021s Metrics: {"bytes_written":6778802,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1057055,"cfile_cache_miss":300,"cfile_cache_miss_bytes":16661407,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":286,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":760,"drs_written":1,"lbm_read_time_us":12561,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":318,"lbm_write_time_us":20224,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":301,"num_input_rowsets":2,"peak_mem_usage":3867322,"rows_written":242700,"spinlock_wait_cycles":768}
I20251026 03:12:30.443583 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2681238 bytes on disk
I20251026 03:12:30.443828 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.444492 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.463946 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":327860,"cfile_init":1,"dirs.queue_time_us":27,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":744,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11400}
I20251026 03:12:30.464126 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1063590 bytes of WAL
I20251026 03:12:30.464205 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:30.464241 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000008 (ops 4397-5018)
I20251026 03:12:30.464623 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:30.464731 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012215
I20251026 03:12:30.472994 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.013s sys 0.006s Metrics: {"bytes_written":450156,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":2670,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1044,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15650}
I20251026 03:12:30.473194 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 147156 bytes on disk
I20251026 03:12:30.473464 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.473618 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012220
I20251026 03:12:30.523532 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.309s user 0.231s sys 0.027s Metrics: {"bytes_written":6925562,"cfile_cache_hit":12,"cfile_cache_hit_bytes":643447,"cfile_cache_miss":308,"cfile_cache_miss_bytes":17463529,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":61,"dirs.run_wall_time_us":19442,"drs_written":1,"lbm_read_time_us":9019,"lbm_reads_lt_1ms":332,"lbm_write_time_us":12660,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":309,"num_input_rowsets":2,"peak_mem_usage":3954532,"rows_written":247950,"spinlock_wait_cycles":1920}
I20251026 03:12:30.523777 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2741919 bytes on disk
I20251026 03:12:30.523917 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.524068 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.544133 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.020s user 0.017s sys 0.000s Metrics: {"bytes_written":369165,"cfile_init":1,"dirs.queue_time_us":21,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1336,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12800}
I20251026 03:12:30.544333 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1062601 bytes of WAL
I20251026 03:12:30.544807 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:30.544885 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000008 (ops 4389-5018)
I20251026 03:12:30.545238 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:30.545352 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012186
I20251026 03:12:30.761242 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.296s user 0.248s sys 0.016s Metrics: {"bytes_written":7200995,"cfile_cache_hit":12,"cfile_cache_hit_bytes":551367,"cfile_cache_miss":320,"cfile_cache_miss_bytes":18289789,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":408,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":949,"drs_written":1,"lbm_read_time_us":8330,"lbm_reads_lt_1ms":344,"lbm_write_time_us":12375,"lbm_writes_lt_1ms":322,"num_input_rowsets":2,"peak_mem_usage":4116105,"rows_written":257900,"spinlock_wait_cycles":809216}
I20251026 03:12:30.761610 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2857428 bytes on disk
I20251026 03:12:30.761830 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.782207 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.811419 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.029s user 0.013s sys 0.000s Metrics: {"bytes_written":272502,"cfile_init":1,"dirs.queue_time_us":34,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":497,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1005,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9450}
I20251026 03:12:30.811898 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 86979 bytes on disk
I20251026 03:12:30.812319 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.812816 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012014
I20251026 03:12:30.855135 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.310s user 0.250s sys 0.031s Metrics: {"bytes_written":7282713,"cfile_cache_hit":11,"cfile_cache_hit_bytes":546985,"cfile_cache_miss":321,"cfile_cache_miss_bytes":18501941,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":752,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":572,"drs_written":1,"lbm_read_time_us":9212,"lbm_reads_lt_1ms":345,"lbm_write_time_us":14437,"lbm_writes_lt_1ms":326,"num_input_rowsets":2,"peak_mem_usage":4159365,"rows_written":260750,"thread_start_us":73,"threads_started":1}
I20251026 03:12:30.855413 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2890292 bytes on disk
I20251026 03:12:30.855552 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:30.863408 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.390s user 0.242s sys 0.021s Metrics: {"bytes_written":7217306,"cfile_cache_hit":10,"cfile_cache_hit_bytes":527517,"cfile_cache_miss":324,"cfile_cache_miss_bytes":18333367,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":427,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":787,"drs_written":1,"lbm_read_time_us":11424,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":343,"lbm_write_time_us":15455,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":322,"num_input_rowsets":2,"peak_mem_usage":4117513,"rows_written":258350}
I20251026 03:12:30.863699 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.875946 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:30.884865 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":331628,"cfile_init":1,"dirs.queue_time_us":17,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":765,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11550,"spinlock_wait_cycles":768}
I20251026 03:12:30.885090 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1051668 bytes of WAL
I20251026 03:12:30.885213 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:30.885334 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000009 (ops 4702-5280)
I20251026 03:12:30.886147 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:30.886276 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2969553 bytes on disk
I20251026 03:12:30.886651 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":57,"lbm_reads_lt_1ms":8}
I20251026 03:12:30.886792 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012124
I20251026 03:12:30.893339 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.017s user 0.012s sys 0.000s Metrics: {"bytes_written":273760,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":82,"dirs.run_wall_time_us":479,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":672,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20251026 03:12:30.893514 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012119
I20251026 03:12:31.223291 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.410s user 0.256s sys 0.016s Metrics: {"bytes_written":7464254,"cfile_cache_hit":12,"cfile_cache_hit_bytes":638590,"cfile_cache_miss":330,"cfile_cache_miss_bytes":18902174,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":229,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":1094,"drs_written":1,"lbm_read_time_us":12661,"lbm_reads_lt_1ms":350,"lbm_write_time_us":15277,"lbm_writes_lt_1ms":336,"num_input_rowsets":2,"peak_mem_usage":4269957,"rows_written":267350,"spinlock_wait_cycles":384}
I20251026 03:12:31.223634 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.239830 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.346s user 0.279s sys 0.020s Metrics: {"bytes_written":7547217,"cfile_cache_hit":11,"cfile_cache_hit_bytes":474032,"cfile_cache_miss":335,"cfile_cache_miss_bytes":19282856,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":648,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":1015,"drs_written":1,"lbm_read_time_us":9524,"lbm_reads_lt_1ms":359,"lbm_write_time_us":15369,"lbm_writes_lt_1ms":339,"num_input_rowsets":2,"peak_mem_usage":4317030,"rows_written":270250}
I20251026 03:12:31.240669 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.247612 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.361s user 0.264s sys 0.017s Metrics: {"bytes_written":7533189,"cfile_cache_hit":11,"cfile_cache_hit_bytes":649909,"cfile_cache_miss":333,"cfile_cache_miss_bytes":19075573,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":740,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":688,"drs_written":1,"lbm_read_time_us":8779,"lbm_reads_lt_1ms":353,"lbm_write_time_us":14133,"lbm_writes_lt_1ms":338,"num_input_rowsets":2,"peak_mem_usage":4307645,"rows_written":269900}
I20251026 03:12:31.248175 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 2995810 bytes on disk
I20251026 03:12:31.248353 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:31.248485 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.257988 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.033s user 0.017s sys 0.000s Metrics: {"bytes_written":377021,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13100}
I20251026 03:12:31.258154 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.012007
I20251026 03:12:31.258845 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":306975,"cfile_init":1,"dirs.queue_time_us":30,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10600}
I20251026 03:12:31.259069 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011941
I20251026 03:12:31.264416 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":319969,"cfile_init":1,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":698,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11100,"thread_start_us":100,"threads_started":1}
I20251026 03:12:31.264604 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011941
I20251026 03:12:31.598829 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.340s user 0.278s sys 0.027s Metrics: {"bytes_written":7839071,"cfile_cache_hit":11,"cfile_cache_hit_bytes":498549,"cfile_cache_miss":349,"cfile_cache_miss_bytes":20038223,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":711,"drs_written":1,"lbm_read_time_us":9832,"lbm_reads_lt_1ms":377,"lbm_write_time_us":16104,"lbm_writes_lt_1ms":352,"num_input_rowsets":2,"peak_mem_usage":4488171,"rows_written":280850,"spinlock_wait_cycles":154240}
I20251026 03:12:31.599277 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3121915 bytes on disk
I20251026 03:12:31.603133 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20251026 03:12:31.604074 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.631541 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.013s sys 0.000s Metrics: {"bytes_written":277748,"cfile_init":1,"dirs.queue_time_us":24,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":894,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1643,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9650}
I20251026 03:12:31.631723 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1058179 bytes of WAL
I20251026 03:12:31.631834 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:31.631916 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000009 (ops 5019-5624)
I20251026 03:12:31.633196 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:31.633883 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011828
I20251026 03:12:31.686882 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.422s user 0.266s sys 0.027s Metrics: {"bytes_written":7843190,"cfile_cache_hit":11,"cfile_cache_hit_bytes":509704,"cfile_cache_miss":351,"cfile_cache_miss_bytes":20036094,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":841,"drs_written":1,"lbm_read_time_us":13536,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":374,"lbm_write_time_us":14913,"lbm_writes_lt_1ms":352,"num_input_rowsets":2,"peak_mem_usage":4491754,"rows_written":281000,"spinlock_wait_cycles":2944}
I20251026 03:12:31.689188 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3123950 bytes on disk
I20251026 03:12:31.689414 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:31.689577 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.727058 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.037s user 0.015s sys 0.004s Metrics: {"bytes_written":391360,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":920,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":891,"lbm_writes_lt_1ms":36,"peak_mem_usage":0,"rows_written":13650}
I20251026 03:12:31.727526 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1069696 bytes of WAL
I20251026 03:12:31.727672 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:31.727772 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000010 (ops 5281-5869)
I20251026 03:12:31.729171 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:31.729323 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 127552 bytes on disk
I20251026 03:12:31.729498 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:31.729923 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011806
I20251026 03:12:31.736292 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.478s user 0.283s sys 0.009s Metrics: {"bytes_written":7829392,"cfile_cache_hit":10,"cfile_cache_hit_bytes":513724,"cfile_cache_miss":352,"cfile_cache_miss_bytes":19988990,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":78,"dirs.run_wall_time_us":7651,"drs_written":1,"lbm_read_time_us":21836,"lbm_reads_1-10_ms":5,"lbm_reads_lt_1ms":375,"lbm_write_time_us":20644,"lbm_writes_lt_1ms":351,"num_input_rowsets":2,"peak_mem_usage":4479168,"rows_written":280450,"spinlock_wait_cycles":7640832,"thread_start_us":117,"threads_started":4}
I20251026 03:12:31.736608 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3117449 bytes on disk
I20251026 03:12:31.736783 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:31.737056 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.771732 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.035s user 0.015s sys 0.005s Metrics: {"bytes_written":455131,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":678,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1155,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15850}
I20251026 03:12:31.772208 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1058117 bytes of WAL
I20251026 03:12:31.772333 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:31.772401 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000009 (ops 5019-5620)
I20251026 03:12:31.772810 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:31.773021 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 149138 bytes on disk
I20251026 03:12:31.773154 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:31.773285 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011790
I20251026 03:12:31.947847 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.314s user 0.293s sys 0.012s Metrics: {"bytes_written":8107093,"cfile_cache_hit":13,"cfile_cache_hit_bytes":807760,"cfile_cache_miss":361,"cfile_cache_miss_bytes":20437022,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1373,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":705,"drs_written":1,"lbm_read_time_us":8076,"lbm_reads_lt_1ms":385,"lbm_write_time_us":13549,"lbm_writes_lt_1ms":362,"mutex_wait_us":2,"num_input_rowsets":2,"peak_mem_usage":4644270,"rows_written":290500,"spinlock_wait_cycles":1536}
I20251026 03:12:31.948168 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:31.962489 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":342428,"cfile_init":1,"dirs.queue_time_us":198,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11950}
I20251026 03:12:31.962702 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3345376 bytes on disk
I20251026 03:12:31.963076 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":29,"lbm_reads_lt_1ms":8}
I20251026 03:12:31.963241 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011709
I20251026 03:12:32.208958 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.479s user 0.291s sys 0.022s Metrics: {"bytes_written":8222139,"cfile_cache_hit":12,"cfile_cache_hit_bytes":730980,"cfile_cache_miss":366,"cfile_cache_miss_bytes":20814272,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":742,"drs_written":1,"lbm_read_time_us":15238,"lbm_reads_lt_1ms":386,"lbm_write_time_us":18757,"lbm_writes_lt_1ms":367,"num_input_rowsets":2,"peak_mem_usage":4709346,"rows_written":294650,"spinlock_wait_cycles":768,"thread_start_us":384,"threads_started":5}
I20251026 03:12:32.209524 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:32.250552 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.041s user 0.012s sys 0.008s Metrics: {"bytes_written":436123,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":702,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1291,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":15250}
I20251026 03:12:32.250870 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011738
I20251026 03:12:32.281526 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.318s user 0.285s sys 0.031s Metrics: {"bytes_written":8437939,"cfile_cache_hit":13,"cfile_cache_hit_bytes":671229,"cfile_cache_miss":373,"cfile_cache_miss_bytes":21458757,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":465,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":14317,"drs_written":1,"lbm_read_time_us":10177,"lbm_reads_lt_1ms":393,"lbm_write_time_us":13691,"lbm_writes_lt_1ms":375,"num_input_rowsets":2,"peak_mem_usage":4837794,"rows_written":302450}
I20251026 03:12:32.281813 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3371202 bytes on disk
I20251026 03:12:32.281984 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:32.282124 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:32.296811 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.015s user 0.008s sys 0.004s Metrics: {"bytes_written":273732,"cfile_init":1,"dirs.queue_time_us":22,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500}
I20251026 03:12:32.297082 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1061233 bytes of WAL
I20251026 03:12:32.297173 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:32.297209 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000010 (ops 5625-6216)
I20251026 03:12:32.297631 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:32.297806 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 88481 bytes on disk
I20251026 03:12:32.297968 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20251026 03:12:32.298504 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011497
I20251026 03:12:32.323185 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.550s user 0.300s sys 0.031s Metrics: {"bytes_written":8268269,"cfile_cache_hit":11,"cfile_cache_hit_bytes":575941,"cfile_cache_miss":369,"cfile_cache_miss_bytes":21082941,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":841,"drs_written":1,"lbm_read_time_us":15794,"lbm_reads_lt_1ms":389,"lbm_write_time_us":27984,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":368,"num_input_rowsets":2,"peak_mem_usage":4732310,"rows_written":296300,"spinlock_wait_cycles":384,"thread_start_us":196,"threads_started":5}
I20251026 03:12:32.323405 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:32.380620 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.057s user 0.016s sys 0.004s Metrics: {"bytes_written":492074,"cfile_init":1,"dirs.queue_time_us":418,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1518,"lbm_writes_lt_1ms":41,"peak_mem_usage":0,"rows_written":17150}
I20251026 03:12:32.382333 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1060046 bytes of WAL
I20251026 03:12:32.382465 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:32.382522 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000010 (ops 5621-6214)
I20251026 03:12:32.383605 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:32.384296 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3465482 bytes on disk
I20251026 03:12:32.385327 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":44,"lbm_reads_lt_1ms":8}
I20251026 03:12:32.385463 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011700
I20251026 03:12:32.628329 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.330s user 0.310s sys 0.007s Metrics: {"bytes_written":8702196,"cfile_cache_hit":13,"cfile_cache_hit_bytes":803214,"cfile_cache_miss":383,"cfile_cache_miss_bytes":22030254,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2220,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":766,"drs_written":1,"lbm_read_time_us":9039,"lbm_reads_lt_1ms":403,"lbm_write_time_us":13533,"lbm_writes_lt_1ms":387,"num_input_rowsets":2,"peak_mem_usage":4994610,"rows_written":311950,"spinlock_wait_cycles":1536}
I20251026 03:12:32.628659 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:32.646123 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.017s user 0.016s sys 0.001s Metrics: {"bytes_written":366719,"cfile_init":1,"dirs.queue_time_us":21,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":739,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":12700,"spinlock_wait_cycles":896}
I20251026 03:12:32.646377 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011559
I20251026 03:12:32.805781 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.420s user 0.305s sys 0.019s Metrics: {"bytes_written":8744321,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1408708,"cfile_cache_miss":381,"cfile_cache_miss_bytes":21517226,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":167,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":773,"drs_written":1,"lbm_read_time_us":12739,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":400,"lbm_write_time_us":15429,"lbm_writes_lt_1ms":389,"num_input_rowsets":2,"peak_mem_usage":5011332,"rows_written":313450,"spinlock_wait_cycles":768}
I20251026 03:12:32.807729 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:32.842103 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.034s user 0.024s sys 0.000s Metrics: {"bytes_written":500719,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":544,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1714,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17500}
I20251026 03:12:32.842360 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011495
I20251026 03:12:32.878451 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.627s user 0.301s sys 0.035s Metrics: {"bytes_written":8645314,"cfile_cache_hit":14,"cfile_cache_hit_bytes":901590,"cfile_cache_miss":382,"cfile_cache_miss_bytes":21768202,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":15908,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":819,"drs_written":1,"lbm_read_time_us":13717,"lbm_reads_lt_1ms":410,"lbm_write_time_us":34382,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":380,"num_input_rowsets":2,"peak_mem_usage":4953893,"rows_written":309900,"spinlock_wait_cycles":4180608,"thread_start_us":66,"threads_started":1}
I20251026 03:12:32.879086 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:32.926409 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.047s user 0.027s sys 0.001s Metrics: {"bytes_written":652312,"cfile_init":1,"dirs.queue_time_us":23,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":756,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2611,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":22850}
I20251026 03:12:32.926983 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011479
I20251026 03:12:33.003603 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.357s user 0.297s sys 0.036s Metrics: {"bytes_written":9051015,"cfile_cache_hit":12,"cfile_cache_hit_bytes":709697,"cfile_cache_miss":398,"cfile_cache_miss_bytes":23053509,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":1067,"drs_written":1,"lbm_read_time_us":9588,"lbm_reads_lt_1ms":426,"lbm_write_time_us":15532,"lbm_writes_lt_1ms":401,"mutex_wait_us":17,"num_input_rowsets":2,"peak_mem_usage":5197768,"rows_written":324650,"spinlock_wait_cycles":768}
I20251026 03:12:33.003852 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:33.020051 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.016s user 0.007s sys 0.008s Metrics: {"bytes_written":325380,"cfile_init":1,"dirs.queue_time_us":47,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":634,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11300}
I20251026 03:12:33.020392 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3733605 bytes on disk
I20251026 03:12:33.020634 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":33,"lbm_reads_lt_1ms":8}
I20251026 03:12:33.021091 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011160
I20251026 03:12:33.226562 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.384s user 0.334s sys 0.011s Metrics: {"bytes_written":9229733,"cfile_cache_hit":12,"cfile_cache_hit_bytes":744623,"cfile_cache_miss":408,"cfile_cache_miss_bytes":23466637,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":648,"drs_written":1,"lbm_read_time_us":9853,"lbm_reads_lt_1ms":436,"lbm_write_time_us":15482,"lbm_writes_lt_1ms":410,"num_input_rowsets":2,"peak_mem_usage":5295331,"rows_written":330950,"spinlock_wait_cycles":1152}
I20251026 03:12:33.226873 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3701955 bytes on disk
I20251026 03:12:33.227023 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:33.227182 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:33.243897 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.017s user 0.009s sys 0.005s Metrics: {"bytes_written":317530,"cfile_init":1,"dirs.queue_time_us":26,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":660,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11000}
I20251026 03:12:33.244189 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 103337 bytes on disk
I20251026 03:12:33.244366 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20251026 03:12:33.244853 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011091
I20251026 03:12:33.488415 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.561s user 0.338s sys 0.035s Metrics: {"bytes_written":9278730,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1252175,"cfile_cache_miss":405,"cfile_cache_miss_bytes":23080205,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":94,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":748,"drs_written":1,"lbm_read_time_us":15018,"lbm_reads_lt_1ms":433,"lbm_write_time_us":24399,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":412,"num_input_rowsets":2,"peak_mem_usage":5319224,"rows_written":332750,"thread_start_us":198,"threads_started":5}
I20251026 03:12:33.489053 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1140582 bytes of WAL
I20251026 03:12:33.489199 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:33.489387 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000011 (ops 5870-6497)
I20251026 03:12:33.490541 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:33.490783 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3722772 bytes on disk
I20251026 03:12:33.490949 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:33.491091 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:33.522512 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.031s user 0.016s sys 0.007s Metrics: {"bytes_written":480325,"cfile_init":1,"dirs.queue_time_us":1847,"dirs.run_cpu_time_us":61,"dirs.run_wall_time_us":465,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1024,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16850}
I20251026 03:12:33.522747 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 161159 bytes on disk
I20251026 03:12:33.523003 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:33.523156 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011160
I20251026 03:12:33.560498 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.539s user 0.330s sys 0.023s Metrics: {"bytes_written":9365735,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1004051,"cfile_cache_miss":410,"cfile_cache_miss_bytes":23591385,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":690,"drs_written":1,"lbm_read_time_us":18192,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":429,"lbm_write_time_us":20693,"lbm_writes_lt_1ms":417,"num_input_rowsets":2,"peak_mem_usage":5382782,"rows_written":335950,"spinlock_wait_cycles":72448,"thread_start_us":201,"threads_started":5}
I20251026 03:12:33.560797 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3759610 bytes on disk
I20251026 03:12:33.560945 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:33.561103 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:33.585588 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.024s user 0.018s sys 0.000s Metrics: {"bytes_written":443968,"cfile_init":1,"dirs.queue_time_us":20,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":826,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":15400}
I20251026 03:12:33.585861 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1056607 bytes of WAL
I20251026 03:12:33.586477 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:33.586611 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000011 (ops 6217-6820)
I20251026 03:12:33.587167 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:33.587309 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011113
I20251026 03:12:33.665693 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.421s user 0.307s sys 0.053s Metrics: {"bytes_written":9530512,"cfile_cache_hit":11,"cfile_cache_hit_bytes":508152,"cfile_cache_miss":423,"cfile_cache_miss_bytes":24539388,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":136,"dirs.run_cpu_time_us":1681,"dirs.run_wall_time_us":13024,"drs_written":1,"lbm_read_time_us":12750,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":442,"lbm_write_time_us":16111,"lbm_writes_lt_1ms":423,"num_input_rowsets":2,"peak_mem_usage":5480574,"rows_written":341950,"spinlock_wait_cycles":384}
I20251026 03:12:33.666476 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:33.718437 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.052s user 0.019s sys 0.001s Metrics: {"bytes_written":410907,"cfile_init":1,"dirs.queue_time_us":97,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":573,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3014,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":14250}
I20251026 03:12:33.718653 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1116430 bytes of WAL
I20251026 03:12:33.719398 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:33.719473 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000011 (ops 6215-6853)
I20251026 03:12:33.719985 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:33.720160 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3963410 bytes on disk
I20251026 03:12:33.720512 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":30,"lbm_reads_lt_1ms":8}
I20251026 03:12:33.720664 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011100
I20251026 03:12:34.032670 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.509s user 0.370s sys 0.007s Metrics: {"bytes_written":9743193,"cfile_cache_hit":12,"cfile_cache_hit_bytes":742547,"cfile_cache_miss":432,"cfile_cache_miss_bytes":24852913,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":114,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":708,"drs_written":1,"lbm_read_time_us":14107,"lbm_reads_lt_1ms":452,"lbm_write_time_us":22752,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":431,"num_input_rowsets":2,"peak_mem_usage":5597379,"rows_written":349600,"spinlock_wait_cycles":1152,"thread_start_us":107,"threads_started":2}
I20251026 03:12:34.032860 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:34.116096 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.083s user 0.026s sys 0.000s Metrics: {"bytes_written":643113,"cfile_init":1,"dirs.queue_time_us":209,"dirs.run_cpu_time_us":60,"dirs.run_wall_time_us":990,"drs_written":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1382,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":22500}
I20251026 03:12:34.116340 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1060668 bytes of WAL
I20251026 03:12:34.116459 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:34.116598 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000012 (ops 6498-7081)
I20251026 03:12:34.116963 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:34.117061 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4133220 bytes on disk
I20251026 03:12:34.117237 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":30,"lbm_reads_lt_1ms":8}
I20251026 03:12:34.118543 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.011048
I20251026 03:12:34.155084 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.564s user 0.332s sys 0.044s Metrics: {"bytes_written":9791667,"cfile_cache_hit":13,"cfile_cache_hit_bytes":1018129,"cfile_cache_miss":435,"cfile_cache_miss_bytes":24705057,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":100,"dirs.run_wall_time_us":5716,"drs_written":1,"lbm_read_time_us":18430,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":456,"lbm_write_time_us":26450,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":432,"num_input_rowsets":2,"peak_mem_usage":5627257,"rows_written":351350,"spinlock_wait_cycles":1488000,"thread_start_us":181,"threads_started":5}
I20251026 03:12:34.155510 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 3938358 bytes on disk
I20251026 03:12:34.155898 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:34.156579 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:34.199141 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.042s user 0.025s sys 0.004s Metrics: {"bytes_written":748927,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":630,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1743,"lbm_writes_lt_1ms":51,"peak_mem_usage":0,"rows_written":26250}
I20251026 03:12:34.200003 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1079512 bytes of WAL
I20251026 03:12:34.200196 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:34.200316 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000012 (ops 6821-7451)
I20251026 03:12:34.200958 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:34.201375 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010992
I20251026 03:12:34.237082 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.514s user 0.332s sys 0.042s Metrics: {"bytes_written":9927554,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1060159,"cfile_cache_miss":438,"cfile_cache_miss_bytes":25026027,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":479,"drs_written":1,"lbm_read_time_us":17953,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":456,"lbm_write_time_us":17146,"lbm_writes_lt_1ms":440,"num_input_rowsets":2,"peak_mem_usage":5705566,"rows_written":356200,"spinlock_wait_cycles":1536,"thread_start_us":222,"threads_started":5}
I20251026 03:12:34.237686 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:34.269697 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.032s user 0.025s sys 0.000s Metrics: {"bytes_written":697975,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":498,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2974,"lbm_writes_lt_1ms":49,"peak_mem_usage":0,"rows_written":24450}
I20251026 03:12:34.270020 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4231918 bytes on disk
I20251026 03:12:34.270210 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20251026 03:12:34.270591 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1179770 bytes of WAL
I20251026 03:12:34.270716 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:34.270833 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000012 (ops 6854-7544)
I20251026 03:12:34.271380 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:34.271502 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010952
I20251026 03:12:34.555573 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.437s user 0.352s sys 0.034s Metrics: {"bytes_written":10366685,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1503118,"cfile_cache_miss":454,"cfile_cache_miss_bytes":25730078,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":120,"dirs.run_wall_time_us":674,"drs_written":1,"lbm_read_time_us":12544,"lbm_reads_lt_1ms":474,"lbm_write_time_us":24975,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":457,"num_input_rowsets":2,"peak_mem_usage":5960619,"rows_written":372100,"thread_start_us":67,"threads_started":2}
I20251026 03:12:34.555896 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:34.578714 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.023s user 0.019s sys 0.002s Metrics: {"bytes_written":485463,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":68,"dirs.run_wall_time_us":2580,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1091,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":17050}
I20251026 03:12:34.579111 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010906
I20251026 03:12:34.682137 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.480s user 0.365s sys 0.025s Metrics: {"bytes_written":10514802,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1580063,"cfile_cache_miss":459,"cfile_cache_miss_bytes":26055099,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":543,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":534,"drs_written":1,"lbm_read_time_us":13848,"lbm_reads_lt_1ms":483,"lbm_write_time_us":27423,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":464,"num_input_rowsets":2,"peak_mem_usage":6042929,"rows_written":377600,"spinlock_wait_cycles":2688,"thread_start_us":43,"threads_started":1}
I20251026 03:12:34.682334 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:34.707911 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.025s user 0.018s sys 0.001s Metrics: {"bytes_written":405915,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":76,"dirs.run_wall_time_us":508,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":14050}
I20251026 03:12:34.708114 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4372480 bytes on disk
I20251026 03:12:34.708297 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":31,"lbm_reads_lt_1ms":8}
I20251026 03:12:34.708410 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010879
I20251026 03:12:34.716470 31233 test_util.cc:276] Using random seed: 662248244
I20251026 03:12:34.721887 31233 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20251026 03:12:34.842520 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.571s user 0.358s sys 0.035s Metrics: {"bytes_written":10601685,"cfile_cache_hit":13,"cfile_cache_hit_bytes":925009,"cfile_cache_miss":465,"cfile_cache_miss_bytes":26942851,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":10092,"drs_written":1,"lbm_read_time_us":18046,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":483,"lbm_write_time_us":24039,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":466,"num_input_rowsets":2,"peak_mem_usage":6095944,"rows_written":380650,"thread_start_us":185,"threads_started":5}
I20251026 03:12:34.842734 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4276367 bytes on disk
I20251026 03:12:34.845016 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:34.845189 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:34.982831 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.136s user 0.023s sys 0.000s Metrics: {"bytes_written":484072,"cfile_init":1,"dirs.queue_time_us":140,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":1010,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":95989,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":38,"peak_mem_usage":0,"rows_written":17000}
I20251026 03:12:34.983089 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010702
I20251026 03:12:35.084455 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.505s user 0.357s sys 0.043s Metrics: {"bytes_written":10837452,"cfile_cache_hit":12,"cfile_cache_hit_bytes":742111,"cfile_cache_miss":476,"cfile_cache_miss_bytes":27762019,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":422,"drs_written":1,"lbm_read_time_us":14727,"lbm_reads_lt_1ms":504,"lbm_write_time_us":20732,"lbm_writes_lt_1ms":478,"num_input_rowsets":2,"peak_mem_usage":6241469,"rows_written":389150,"spinlock_wait_cycles":3143168,"thread_start_us":189,"threads_started":5}
I20251026 03:12:35.084648 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1062315 bytes of WAL
I20251026 03:12:35.084748 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:35.084789 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000013 (ops 7082-7666)
I20251026 03:12:35.085125 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:35.085216 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:35.112860 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.018s sys 0.004s Metrics: {"bytes_written":573758,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":785,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1070,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":20150}
I20251026 03:12:35.113152 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4566695 bytes on disk
I20251026 03:12:35.113453 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":29,"lbm_reads_lt_1ms":8}
I20251026 03:12:35.114230 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010638
I20251026 03:12:35.210585 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.502s user 0.350s sys 0.052s Metrics: {"bytes_written":10906888,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1174520,"cfile_cache_miss":480,"cfile_cache_miss_bytes":27515434,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":86,"dirs.run_wall_time_us":571,"drs_written":1,"lbm_read_time_us":12939,"lbm_reads_lt_1ms":500,"lbm_write_time_us":21682,"lbm_writes_lt_1ms":482,"num_input_rowsets":2,"peak_mem_usage":6285088,"rows_written":391650,"spinlock_wait_cycles":3584,"thread_start_us":238,"threads_started":5}
I20251026 03:12:35.210954 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4404333 bytes on disk
I20251026 03:12:35.211362 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4}
I20251026 03:12:35.211529 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:35.246130 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.035s user 0.015s sys 0.010s Metrics: {"bytes_written":601361,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":74,"dirs.run_wall_time_us":491,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2682,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":21050}
I20251026 03:12:35.246706 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1100620 bytes of WAL
I20251026 03:12:35.246799 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:35.246845 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000013 (ops 7452-8092)
I20251026 03:12:35.247211 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:35.247359 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010602
I20251026 03:12:35.545735 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.563s user 0.394s sys 0.035s Metrics: {"bytes_written":11071789,"cfile_cache_hit":11,"cfile_cache_hit_bytes":580509,"cfile_cache_miss":487,"cfile_cache_miss_bytes":28543459,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":285,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":711,"drs_written":1,"lbm_read_time_us":15638,"lbm_reads_lt_1ms":511,"lbm_write_time_us":23592,"lbm_writes_lt_1ms":489,"num_input_rowsets":2,"peak_mem_usage":6379635,"rows_written":397650,"spinlock_wait_cycles":768,"thread_start_us":227,"threads_started":5}
I20251026 03:12:35.545940 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:35.579952 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.034s user 0.022s sys 0.003s Metrics: {"bytes_written":651025,"cfile_init":1,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1782,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":22800}
I20251026 03:12:35.580130 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1064827 bytes of WAL
I20251026 03:12:35.580209 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:35.580238 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000013 (ops 7545-8176)
I20251026 03:12:35.580607 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:35.580716 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010521
I20251026 03:12:35.626546 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.512s user 0.403s sys 0.018s Metrics: {"bytes_written":11394657,"cfile_cache_hit":12,"cfile_cache_hit_bytes":831505,"cfile_cache_miss":500,"cfile_cache_miss_bytes":29145477,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":740,"drs_written":1,"lbm_read_time_us":13690,"lbm_reads_lt_1ms":520,"lbm_write_time_us":21121,"lbm_writes_lt_1ms":503,"num_input_rowsets":2,"peak_mem_usage":6564132,"rows_written":409300,"spinlock_wait_cycles":2176,"thread_start_us":175,"threads_started":5}
I20251026 03:12:35.627357 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:35.655023 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.015s sys 0.004s Metrics: {"bytes_written":423486,"cfile_init":1,"dirs.queue_time_us":835,"dirs.run_cpu_time_us":55,"dirs.run_wall_time_us":2611,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2687,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":37,"peak_mem_usage":0,"rows_written":14750}
I20251026 03:12:35.655545 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1054988 bytes of WAL
I20251026 03:12:35.655691 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:35.656121 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000014 (ops 7667-8247)
I20251026 03:12:35.656554 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:35.656693 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4746670 bytes on disk
I20251026 03:12:35.657037 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":36,"lbm_reads_lt_1ms":8}
I20251026 03:12:35.657344 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010313
I20251026 03:12:35.755807 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.508s user 0.390s sys 0.037s Metrics: {"bytes_written":11489775,"cfile_cache_hit":11,"cfile_cache_hit_bytes":592302,"cfile_cache_miss":505,"cfile_cache_miss_bytes":29631218,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":449,"drs_written":1,"lbm_read_time_us":18183,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":527,"lbm_write_time_us":21127,"lbm_writes_lt_1ms":507,"num_input_rowsets":2,"peak_mem_usage":6618475,"rows_written":412700,"spinlock_wait_cycles":1536,"thread_start_us":177,"threads_started":5}
I20251026 03:12:35.756091 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4646911 bytes on disk
I20251026 03:12:35.756495 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":25,"lbm_reads_lt_1ms":4}
I20251026 03:12:35.756661 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:35.801913 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.045s user 0.011s sys 0.011s Metrics: {"bytes_written":479122,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":469,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1176,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16800}
I20251026 03:12:35.802155 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010290
I20251026 03:12:36.148918 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.491s user 0.428s sys 0.012s Metrics: {"bytes_written":11800585,"cfile_cache_hit":14,"cfile_cache_hit_bytes":1055214,"cfile_cache_miss":518,"cfile_cache_miss_bytes":30021414,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":143,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":805,"drs_written":1,"lbm_read_time_us":15624,"lbm_reads_lt_1ms":538,"lbm_write_time_us":19959,"lbm_writes_lt_1ms":520,"num_input_rowsets":2,"peak_mem_usage":6814158,"rows_written":424050,"spinlock_wait_cycles":1152,"thread_start_us":50,"threads_started":1}
I20251026 03:12:36.149212 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4778845 bytes on disk
I20251026 03:12:36.149431 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.149595 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:36.282693 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.702s user 0.420s sys 0.037s Metrics: {"bytes_written":11699597,"cfile_cache_hit":10,"cfile_cache_hit_bytes":529509,"cfile_cache_miss":516,"cfile_cache_miss_bytes":30269561,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":674,"drs_written":1,"lbm_read_time_us":15505,"lbm_reads_lt_1ms":544,"lbm_write_time_us":26061,"lbm_writes_lt_1ms":516,"num_input_rowsets":2,"peak_mem_usage":6744306,"rows_written":420450,"spinlock_wait_cycles":1536,"thread_start_us":204,"threads_started":5}
I20251026 03:12:36.283214 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4737108 bytes on disk
I20251026 03:12:36.283394 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.284353 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:36.289470 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.140s user 0.025s sys 0.000s Metrics: {"bytes_written":479063,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":634,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1225,"lbm_writes_lt_1ms":40,"peak_mem_usage":0,"rows_written":16800,"spinlock_wait_cycles":188032}
I20251026 03:12:36.299132 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 158377 bytes on disk
I20251026 03:12:36.300923 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.302999 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010108
I20251026 03:12:36.343767 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.059s user 0.033s sys 0.002s Metrics: {"bytes_written":664099,"cfile_init":1,"dirs.queue_time_us":19,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":649,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3289,"lbm_writes_lt_1ms":47,"peak_mem_usage":0,"rows_written":23300}
I20251026 03:12:36.344056 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1109200 bytes of WAL
I20251026 03:12:36.344198 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:36.344614 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000014 (ops 8177-8827)
I20251026 03:12:36.345597 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:36.346221 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010139
I20251026 03:12:36.387820 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.586s user 0.398s sys 0.056s Metrics: {"bytes_written":11954554,"cfile_cache_hit":12,"cfile_cache_hit_bytes":740105,"cfile_cache_miss":526,"cfile_cache_miss_bytes":30730417,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":691,"drs_written":1,"lbm_read_time_us":17529,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":547,"lbm_write_time_us":24062,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":526,"num_input_rowsets":2,"peak_mem_usage":6902309,"rows_written":429500,"spinlock_wait_cycles":384,"thread_start_us":688,"threads_started":5}
I20251026 03:12:36.388115 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 4842118 bytes on disk
I20251026 03:12:36.388236 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.388428 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:36.434593 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.046s user 0.021s sys 0.001s Metrics: {"bytes_written":470356,"cfile_init":1,"dirs.queue_time_us":18,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":707,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12353,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":16450}
I20251026 03:12:36.435009 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1069048 bytes of WAL
I20251026 03:12:36.435108 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:36.435154 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000014 (ops 8093-8722)
I20251026 03:12:36.435549 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:36.436056 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 156677 bytes on disk
I20251026 03:12:36.436197 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.436349 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010052
I20251026 03:12:36.826367 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.523s user 0.396s sys 0.048s Metrics: {"bytes_written":12266824,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1219649,"cfile_cache_miss":537,"cfile_cache_miss_bytes":31086933,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":998,"drs_written":1,"lbm_read_time_us":27726,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":556,"lbm_write_time_us":19254,"lbm_writes_lt_1ms":539,"num_input_rowsets":2,"peak_mem_usage":7085083,"rows_written":440850,"spinlock_wait_cycles":4480}
I20251026 03:12:36.826714 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:36.849623 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.023s user 0.021s sys 0.000s Metrics: {"bytes_written":547485,"cfile_init":1,"dirs.queue_time_us":21,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":19150}
I20251026 03:12:36.849906 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5155388 bytes on disk
I20251026 03:12:36.850180 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":27,"lbm_reads_lt_1ms":8}
I20251026 03:12:36.850375 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010146
I20251026 03:12:36.954648 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.608s user 0.402s sys 0.055s Metrics: {"bytes_written":12347217,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1238413,"cfile_cache_miss":539,"cfile_cache_miss_bytes":31267373,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":11456,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":672,"drs_written":1,"lbm_read_time_us":15528,"lbm_reads_lt_1ms":563,"lbm_write_time_us":28277,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":541,"num_input_rowsets":2,"peak_mem_usage":7125704,"rows_written":443750,"thread_start_us":310,"threads_started":6}
I20251026 03:12:36.954927 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5006570 bytes on disk
I20251026 03:12:36.955076 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.955201 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:36.989707 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.034s user 0.010s sys 0.011s Metrics: {"bytes_written":531716,"cfile_init":1,"dirs.queue_time_us":22,"dirs.run_cpu_time_us":80,"dirs.run_wall_time_us":1468,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":988,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18700}
I20251026 03:12:36.990039 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 177452 bytes on disk
I20251026 03:12:36.990190 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:36.990335 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010121
I20251026 03:12:37.071206 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.635s user 0.444s sys 0.033s Metrics: {"bytes_written":12404969,"cfile_cache_hit":16,"cfile_cache_hit_bytes":1306041,"cfile_cache_miss":542,"cfile_cache_miss_bytes":31381263,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":11776,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1065,"drs_written":1,"lbm_read_time_us":18120,"lbm_reads_lt_1ms":562,"lbm_write_time_us":33753,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":541,"num_input_rowsets":2,"peak_mem_usage":7168306,"rows_written":445950,"spinlock_wait_cycles":3072,"thread_start_us":1021,"threads_started":6}
I20251026 03:12:37.071542 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:37.125090 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.053s user 0.022s sys 0.004s Metrics: {"bytes_written":547625,"cfile_init":1,"dirs.queue_time_us":1628,"dirs.run_cpu_time_us":77,"dirs.run_wall_time_us":439,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1438,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":19150}
I20251026 03:12:37.125360 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.010093
I20251026 03:12:37.367157 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.517s user 0.444s sys 0.028s Metrics: {"bytes_written":12797478,"cfile_cache_hit":11,"cfile_cache_hit_bytes":587337,"cfile_cache_miss":561,"cfile_cache_miss_bytes":33126035,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":750,"drs_written":1,"lbm_read_time_us":14007,"lbm_reads_lt_1ms":581,"lbm_write_time_us":19744,"lbm_writes_lt_1ms":564,"num_input_rowsets":2,"peak_mem_usage":7392248,"rows_written":460000,"spinlock_wait_cycles":768,"thread_start_us":178,"threads_started":5}
I20251026 03:12:37.367466 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1053398 bytes of WAL
I20251026 03:12:37.367569 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:37.367628 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000015 (ops 8248-8827)
I20251026 03:12:37.368043 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:37.368366 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:37.396028 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.013s sys 0.005s Metrics: {"bytes_written":437175,"cfile_init":1,"dirs.queue_time_us":50,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":809,"lbm_writes_lt_1ms":39,"peak_mem_usage":0,"rows_written":15300}
I20251026 03:12:37.396287 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5340065 bytes on disk
I20251026 03:12:37.397550 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20251026 03:12:37.397727 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1065924 bytes of WAL
I20251026 03:12:37.397840 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:37.397898 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000016 (ops 8828-9414)
I20251026 03:12:37.398319 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:37.398433 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009722
I20251026 03:12:37.517802 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.527s user 0.439s sys 0.035s Metrics: {"bytes_written":12863033,"cfile_cache_hit":15,"cfile_cache_hit_bytes":1223932,"cfile_cache_miss":561,"cfile_cache_miss_bytes":32667716,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":216,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":658,"drs_written":1,"lbm_read_time_us":12997,"lbm_reads_lt_1ms":581,"lbm_write_time_us":24234,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":566,"num_input_rowsets":2,"peak_mem_usage":7434162,"rows_written":462450,"spinlock_wait_cycles":692608,"thread_start_us":200,"threads_started":5}
I20251026 03:12:37.519255 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5223232 bytes on disk
I20251026 03:12:37.519451 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:37.519589 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:37.549108 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.029s user 0.014s sys 0.008s Metrics: {"bytes_written":533071,"cfile_init":1,"dirs.queue_time_us":41,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18750}
I20251026 03:12:37.549703 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1054862 bytes of WAL
I20251026 03:12:37.549882 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:37.549958 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000015 (ops 8828-9427)
I20251026 03:12:37.550392 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:37.550755 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009704
I20251026 03:12:38.149680 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 1.024s user 0.463s sys 0.039s Metrics: {"bytes_written":12935609,"cfile_cache_hit":17,"cfile_cache_hit_bytes":1511306,"cfile_cache_miss":561,"cfile_cache_miss_bytes":32570022,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":7443,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":2227,"drs_written":1,"lbm_read_time_us":25649,"lbm_reads_1-10_ms":3,"lbm_reads_lt_1ms":586,"lbm_write_time_us":53999,"lbm_writes_1-10_ms":8,"lbm_writes_lt_1ms":561,"num_input_rowsets":2,"peak_mem_usage":7475121,"rows_written":465100,"spinlock_wait_cycles":768,"thread_start_us":256,"threads_started":6}
I20251026 03:12:38.150018 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5253140 bytes on disk
I20251026 03:12:38.151270 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4}
I20251026 03:12:38.151475 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=3.921653
I20251026 03:12:38.216774 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.818s user 0.481s sys 0.012s Metrics: {"bytes_written":13215006,"cfile_cache_hit":12,"cfile_cache_hit_bytes":847983,"cfile_cache_miss":584,"cfile_cache_miss_bytes":34001285,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":15798,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":772,"drs_written":1,"lbm_read_time_us":33483,"lbm_reads_1-10_ms":6,"lbm_reads_lt_1ms":598,"lbm_write_time_us":32863,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":577,"num_input_rowsets":2,"peak_mem_usage":7648006,"rows_written":475300,"spinlock_wait_cycles":3353600,"thread_start_us":281,"threads_started":6}
I20251026 03:12:38.216925 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=2.929465
I20251026 03:12:38.219478 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.669s user 0.467s sys 0.025s Metrics: {"bytes_written":13383041,"cfile_cache_hit":20,"cfile_cache_hit_bytes":2016214,"cfile_cache_miss":582,"cfile_cache_miss_bytes":33259044,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":10648,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":980,"drs_written":1,"lbm_read_time_us":23212,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":604,"lbm_write_time_us":28111,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":587,"num_input_rowsets":2,"peak_mem_usage":7737122,"rows_written":481200,"spinlock_wait_cycles":23424,"thread_start_us":263,"threads_started":6}
I20251026 03:12:38.219640 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5440675 bytes on disk
I20251026 03:12:38.222056 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:38.222175 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:38.227934 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.076s user 0.042s sys 0.000s Metrics: {"bytes_written":1125020,"cfile_init":1,"dirs.queue_time_us":29,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3346,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":65,"peak_mem_usage":0,"rows_written":39600}
I20251026 03:12:38.228145 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 2112877 bytes of WAL
I20251026 03:12:38.228677 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 2 log segments from log reader
I20251026 03:12:38.228724 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000015 (ops 8723-9325)
I20251026 03:12:38.228751 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000016 (ops 9326-9943)
I20251026 03:12:38.233263 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.005s user 0.001s sys 0.000s Metrics: {}
I20251026 03:12:38.233435 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009710
I20251026 03:12:38.319141 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.095s user 0.028s sys 0.008s Metrics: {"bytes_written":739629,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":658,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1537,"lbm_writes_lt_1ms":50,"peak_mem_usage":0,"rows_written":25900}
I20251026 03:12:38.320606 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1078950 bytes of WAL
I20251026 03:12:38.320765 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:38.320854 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000016 (ops 9428-10061)
I20251026 03:12:38.321336 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:38.322299 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009681
I20251026 03:12:38.348970 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.127s user 0.044s sys 0.000s Metrics: {"bytes_written":906797,"cfile_init":1,"dirs.queue_time_us":1719,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":562,"drs_written":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7265,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":55,"peak_mem_usage":0,"rows_written":32000}
I20251026 03:12:38.349118 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1055330 bytes of WAL
I20251026 03:12:38.349180 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:38.349206 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000017 (ops 9415-9995)
I20251026 03:12:38.349622 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:38.349709 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009683
I20251026 03:12:39.108569 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.759s user 0.507s sys 0.054s Metrics: {"bytes_written":14101842,"cfile_cache_hit":20,"cfile_cache_hit_bytes":2050172,"cfile_cache_miss":610,"cfile_cache_miss_bytes":35109810,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":8876,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1921,"drs_written":1,"lbm_read_time_us":23040,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":637,"lbm_write_time_us":32231,"lbm_writes_lt_1ms":618,"num_input_rowsets":2,"peak_mem_usage":8151259,"rows_written":507300,"spinlock_wait_cycles":1920,"thread_start_us":268,"threads_started":6}
I20251026 03:12:39.108917 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5741136 bytes on disk
I20251026 03:12:39.109441 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:39.109620 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=2.929465
I20251026 03:12:39.153738 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.044s user 0.039s sys 0.000s Metrics: {"bytes_written":1008750,"cfile_init":1,"dirs.queue_time_us":456,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":512,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2976,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":60,"peak_mem_usage":0,"rows_written":35600}
I20251026 03:12:39.154057 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1098697 bytes of WAL
I20251026 03:12:39.154188 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:39.154246 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000018 (ops 9996-10600)
I20251026 03:12:39.154644 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:39.154752 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 349694 bytes on disk
I20251026 03:12:39.155252 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20251026 03:12:39.155411 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009290
I20251026 03:12:39.243542 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.921s user 0.531s sys 0.022s Metrics: {"bytes_written":14097008,"cfile_cache_hit":19,"cfile_cache_hit_bytes":1874527,"cfile_cache_miss":609,"cfile_cache_miss_bytes":35292973,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":80521,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":8022,"drs_written":1,"lbm_read_time_us":41891,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":630,"lbm_write_time_us":48579,"lbm_writes_1-10_ms":9,"lbm_writes_lt_1ms":609,"num_input_rowsets":2,"peak_mem_usage":8154117,"rows_written":507100,"spinlock_wait_cycles":768,"thread_start_us":379,"threads_started":6}
I20251026 03:12:39.243809 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 5738940 bytes on disk
I20251026 03:12:39.243990 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4}
I20251026 03:12:39.244122 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=3.921653
I20251026 03:12:39.290246 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.046s user 0.042s sys 0.000s Metrics: {"bytes_written":1109127,"cfile_init":1,"dirs.queue_time_us":29,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":499,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2472,"lbm_writes_lt_1ms":65,"peak_mem_usage":0,"rows_written":39150}
I20251026 03:12:39.290540 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1099108 bytes of WAL
I20251026 03:12:39.290730 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:39.290786 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000017 (ops 10062-10715)
I20251026 03:12:39.291266 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:39.291445 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 386306 bytes on disk
I20251026 03:12:39.291618 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20251026 03:12:39.291850 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009255
I20251026 03:12:39.563942 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 1.329s user 0.533s sys 0.029s Metrics: {"bytes_written":14031578,"cfile_cache_hit":18,"cfile_cache_hit_bytes":1756959,"cfile_cache_miss":608,"cfile_cache_miss_bytes":35214905,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":20488,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":2693,"drs_written":1,"lbm_read_time_us":51416,"lbm_reads_1-10_ms":7,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":624,"lbm_write_time_us":49916,"lbm_writes_1-10_ms":8,"lbm_writes_lt_1ms":608,"num_input_rowsets":2,"peak_mem_usage":8101406,"rows_written":504700,"spinlock_wait_cycles":9056384,"thread_start_us":264,"threads_started":6}
I20251026 03:12:39.564388 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=4.913840
I20251026 03:12:39.661571 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.097s user 0.053s sys 0.001s Metrics: {"bytes_written":1425748,"cfile_init":1,"dirs.queue_time_us":20,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":802,"drs_written":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4325,"lbm_writes_lt_1ms":78,"peak_mem_usage":0,"rows_written":50400}
I20251026 03:12:39.661885 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1100573 bytes of WAL
I20251026 03:12:39.662055 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:39.662140 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000017 (ops 9944-10607)
I20251026 03:12:39.663092 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:39.663239 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 6220357 bytes on disk
I20251026 03:12:39.663457 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20251026 03:12:39.663900 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009163
I20251026 03:12:39.776643 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.621s user 0.524s sys 0.037s Metrics: {"bytes_written":15085984,"cfile_cache_hit":12,"cfile_cache_hit_bytes":791687,"cfile_cache_miss":658,"cfile_cache_miss_bytes":38989703,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":8139,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":1721,"drs_written":1,"lbm_read_time_us":16802,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":677,"lbm_write_time_us":30849,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":660,"num_input_rowsets":2,"peak_mem_usage":8726114,"rows_written":542900,"spinlock_wait_cycles":43136,"thread_start_us":80,"threads_started":1}
I20251026 03:12:39.776854 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:39.805364 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.028s user 0.021s sys 0.000s Metrics: {"bytes_written":498245,"cfile_init":1,"dirs.queue_time_us":23,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1407,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":17400}
I20251026 03:12:39.805562 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 6319705 bytes on disk
I20251026 03:12:39.805732 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":32,"lbm_reads_lt_1ms":8}
I20251026 03:12:39.805878 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009085
I20251026 03:12:39.943657 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.652s user 0.539s sys 0.030s Metrics: {"bytes_written":15178365,"cfile_cache_hit":11,"cfile_cache_hit_bytes":643377,"cfile_cache_miss":663,"cfile_cache_miss_bytes":39381429,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":11340,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":743,"drs_written":1,"lbm_read_time_us":15831,"lbm_reads_lt_1ms":683,"lbm_write_time_us":26901,"lbm_writes_lt_1ms":665,"num_input_rowsets":2,"peak_mem_usage":8775340,"rows_written":546250,"spinlock_wait_cycles":3968,"thread_start_us":280,"threads_started":6}
I20251026 03:12:39.943964 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 6191512 bytes on disk
I20251026 03:12:39.944262 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:39.944425 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.000000
I20251026 03:12:39.978039 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.034s user 0.022s sys 0.000s Metrics: {"bytes_written":514717,"cfile_init":1,"dirs.queue_time_us":176,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1206,"lbm_writes_lt_1ms":42,"peak_mem_usage":0,"rows_written":18050}
I20251026 03:12:39.978421 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.009057
I20251026 03:12:40.373673 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.710s user 0.555s sys 0.050s Metrics: {"bytes_written":15422943,"cfile_cache_hit":11,"cfile_cache_hit_bytes":673602,"cfile_cache_miss":673,"cfile_cache_miss_bytes":39994398,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":115739,"drs_written":1,"lbm_read_time_us":20954,"lbm_reads_lt_1ms":693,"lbm_write_time_us":32230,"lbm_writes_lt_1ms":676,"num_input_rowsets":2,"peak_mem_usage":8909536,"rows_written":555100,"spinlock_wait_cycles":81536,"thread_start_us":57,"threads_started":1}
I20251026 03:12:40.373988 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 6294080 bytes on disk
I20251026 03:12:40.374233 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4}
I20251026 03:12:40.374374 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:40.448150 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.074s user 0.022s sys 0.003s Metrics: {"bytes_written":582786,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":730,"drs_written":1,"lbm_read_time_us":17,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1525,"lbm_writes_lt_1ms":44,"peak_mem_usage":0,"rows_written":20500}
I20251026 03:12:40.448400 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1090479 bytes of WAL
I20251026 03:12:40.449123 9121 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:40.449229 9121 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P e46fbbdc3a05494998b24380ff3e0244: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-1-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000018 (ops 10608-11216)
I20251026 03:12:40.449705 9121 maintenance_manager.cc:643] P e46fbbdc3a05494998b24380ff3e0244: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:40.449855 9181 maintenance_manager.cc:419] P e46fbbdc3a05494998b24380ff3e0244: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.008960
I20251026 03:12:40.496173 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.690s user 0.534s sys 0.035s Metrics: {"bytes_written":15565443,"cfile_cache_hit":13,"cfile_cache_hit_bytes":898451,"cfile_cache_miss":685,"cfile_cache_miss_bytes":40197281,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":41286,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":4477,"drs_written":1,"lbm_read_time_us":17488,"lbm_reads_lt_1ms":705,"lbm_write_time_us":29105,"lbm_writes_lt_1ms":681,"num_input_rowsets":2,"peak_mem_usage":9027070,"rows_written":560300,"spinlock_wait_cycles":2522880,"thread_start_us":55,"threads_started":1}
I20251026 03:12:40.496392 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1051775 bytes of WAL
I20251026 03:12:40.496500 9045 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:40.496548 9045 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P a74cdf9569d5402fa3b5bcc40875eb2f: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-0-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000019 (ops 10601-11179)
I20251026 03:12:40.497347 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:40.497478 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 6354351 bytes on disk
I20251026 03:12:40.497617 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:40.497736 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:40.524566 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.027s user 0.022s sys 0.000s Metrics: {"bytes_written":547623,"cfile_init":1,"dirs.queue_time_us":26,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1093,"lbm_writes_lt_1ms":43,"peak_mem_usage":0,"rows_written":19150}
I20251026 03:12:40.524812 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 182614 bytes on disk
I20251026 03:12:40.524981 9045 maintenance_manager.cc:643] P a74cdf9569d5402fa3b5bcc40875eb2f: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4}
I20251026 03:12:40.525121 9105 maintenance_manager.cc:419] P a74cdf9569d5402fa3b5bcc40875eb2f: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.008918
I20251026 03:12:40.748520 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.770s user 0.530s sys 0.048s Metrics: {"bytes_written":15675418,"cfile_cache_hit":13,"cfile_cache_hit_bytes":899690,"cfile_cache_miss":687,"cfile_cache_miss_bytes":40485254,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":16278,"dirs.run_cpu_time_us":114,"dirs.run_wall_time_us":659,"drs_written":1,"lbm_read_time_us":19685,"lbm_reads_lt_1ms":711,"lbm_write_time_us":35594,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":683,"num_input_rowsets":2,"peak_mem_usage":9088106,"rows_written":564300,"spinlock_wait_cycles":1280,"thread_start_us":351,"threads_started":6}
I20251026 03:12:40.748870 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): 6401077 bytes on disk
I20251026 03:12:40.749138 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: UndoDeltaBlockGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":14,"lbm_reads_lt_1ms":4}
I20251026 03:12:40.749419 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.937278
I20251026 03:12:40.786543 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: FlushMRSOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.037s user 0.020s sys 0.003s Metrics: {"bytes_written":596122,"cfile_init":1,"dirs.queue_time_us":26,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1125,"lbm_writes_lt_1ms":45,"peak_mem_usage":0,"rows_written":20850}
I20251026 03:12:40.786873 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238): free 1053361 bytes of WAL
I20251026 03:12:40.787045 9196 log_reader.cc:385] T 6faa7d3ea6644b65aca9c9fe97dbc238: removed 1 log segments from log reader
I20251026 03:12:40.787118 9196 log.cc:1079] T 6faa7d3ea6644b65aca9c9fe97dbc238 P f11d9d7d1c1446a9a10998d50d86c657: Deleting log segment in path: /tmp/dist-test-taskLKxJmk/test-tmp/txn_participant-itest.0.ShouldGCWals_ParticipantCopyITest.TestCopyParticipantOps_0.1761448064857930-31233-0/minicluster-data/ts-2-root/wals/6faa7d3ea6644b65aca9c9fe97dbc238/wal-000000018 (ops 10716-11298)
I20251026 03:12:40.787813 9196 maintenance_manager.cc:643] P f11d9d7d1c1446a9a10998d50d86c657: LogGCOp(6faa7d3ea6644b65aca9c9fe97dbc238) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20251026 03:12:40.788138 9256 maintenance_manager.cc:419] P f11d9d7d1c1446a9a10998d50d86c657: Scheduling CompactRowSetsOp(6faa7d3ea6644b65aca9c9fe97dbc238): perf score=1.008833
F20251026 03:12:41.059769 9461 test_workload.cc:305] Already present: key already present
*** Check failure stack trace: ***
*** Aborted at 1761448361 (unix time) try "date -d @1761448361" if you are using GNU date ***
F20251026 03:12:41.063155 9460 test_workload.cc:305] Already present: key already present
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800007a01) received by PID 31233 (TID 0x7f046b2c7700) from PID 31233; stack trace: ***
F20251026 03:12:41.063155 9460 test_workload.cc:305] Already present: key already presentF20251026 03:12:41.067440 9459 test_workload.cc:305] Already present: key already present
*** Check failure stack trace: ***
@ 0x7f051172f980 (unknown) at ??:0
F20251026 03:12:41.063155 9460 test_workload.cc:305] Already present: key already presentF20251026 03:12:41.067440 9459 test_workload.cc:305] Already present: key already presentF20251026 03:12:41.074344 9458 test_workload.cc:305] Already present: key already present
*** Check failure stack trace: ***
@ 0x7f050f9defb7 gsignal at ??:0
@ 0x7f050f9e0921 abort at ??:0
@ 0x7f0510847dcd google::LogMessage::Fail() at ??:0
@ 0x7f051084bb93 google::LogMessage::SendToLog() at ??:0
@ 0x7f05108477cc google::LogMessage::Flush() at ??:0
@ 0x7f0510848f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f051208e4a8 kudu::TestWorkload::GetNumberOfErrors() at ??:0
@ 0x7f051209130c kudu::TestWorkload::WriteThread() at ??:0
@ 0x7f05100666df (unknown) at ??:0
@ 0x7f05117246db start_thread at ??:0
@ 0x7f050fac171f clone at ??:0