[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250903 08:11:04.791971 12408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.30.62:43647
I20250903 08:11:04.792883 12408 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250903 08:11:04.793404 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:04.799619 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:04.799772 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:04.799782 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:04.799615 12416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:04.800292 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:04.800387 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:04.800434 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887064800430 us; error 0 us; skew 500 ppm
I20250903 08:11:04.802127 12408 webserver.cc:480] Webserver started at http://127.12.30.62:43567/ using document root <none> and password file <none>
I20250903 08:11:04.802628 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:04.802686 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:04.802904 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:04.804338 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/master-0-root/instance:
uuid: "45f393be2d8c49bb8297ce6b832fac0f"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:04.807638 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250903 08:11:04.809676 12421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:04.810673 12408 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:11:04.810765 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/master-0-root
uuid: "45f393be2d8c49bb8297ce6b832fac0f"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:04.810865 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:04.844623 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:04.845188 12408 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250903 08:11:04.845340 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:04.846161 12427 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20250903 08:11:04.852077 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.62:43647
I20250903 08:11:04.852102 12473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.62:43647 every 8 connection(s)
I20250903 08:11:04.854475 12474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:04.859139 12474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f: Bootstrap starting.
I20250903 08:11:04.861469 12474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:04.862038 12474 log.cc:826] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f: Log is configured to *not* fsync() on all Append() calls
I20250903 08:11:04.863360 12474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f: No bootstrap required, opened a new log
I20250903 08:11:04.865557 12474 raft_consensus.cc:357] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f393be2d8c49bb8297ce6b832fac0f" member_type: VOTER }
I20250903 08:11:04.865689 12474 raft_consensus.cc:383] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:04.865741 12474 raft_consensus.cc:738] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f393be2d8c49bb8297ce6b832fac0f, State: Initialized, Role: FOLLOWER
I20250903 08:11:04.866125 12474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f393be2d8c49bb8297ce6b832fac0f" member_type: VOTER }
I20250903 08:11:04.866233 12474 raft_consensus.cc:397] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:11:04.866289 12474 raft_consensus.cc:491] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:11:04.866385 12474 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:04.867058 12474 raft_consensus.cc:513] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f393be2d8c49bb8297ce6b832fac0f" member_type: VOTER }
I20250903 08:11:04.867316 12474 leader_election.cc:304] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45f393be2d8c49bb8297ce6b832fac0f; no voters:
I20250903 08:11:04.867558 12474 leader_election.cc:290] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:11:04.867638 12477 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:11:04.867872 12477 raft_consensus.cc:695] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 1 LEADER]: Becoming Leader. State: Replica: 45f393be2d8c49bb8297ce6b832fac0f, State: Running, Role: LEADER
I20250903 08:11:04.868160 12477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f393be2d8c49bb8297ce6b832fac0f" member_type: VOTER }
I20250903 08:11:04.868495 12474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:11:04.869341 12478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45f393be2d8c49bb8297ce6b832fac0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f393be2d8c49bb8297ce6b832fac0f" member_type: VOTER } }
I20250903 08:11:04.869714 12478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [sys.catalog]: This master's current role is: LEADER
I20250903 08:11:04.869612 12479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45f393be2d8c49bb8297ce6b832fac0f. Latest consensus state: current_term: 1 leader_uuid: "45f393be2d8c49bb8297ce6b832fac0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45f393be2d8c49bb8297ce6b832fac0f" member_type: VOTER } }
I20250903 08:11:04.870009 12479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [sys.catalog]: This master's current role is: LEADER
I20250903 08:11:04.870193 12486 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:11:04.872622 12486 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:11:04.872942 12408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 08:11:04.877174 12486 catalog_manager.cc:1349] Generated new cluster ID: 08bcc988d6ed44aaaa4768e52767d13e
I20250903 08:11:04.877231 12486 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:11:04.902190 12486 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:11:04.903286 12486 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:11:04.910549 12486 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f: Generated new TSK 0
I20250903 08:11:04.911326 12486 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:11:04.937475 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:04.939494 12496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:04.939637 12499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:04.939664 12497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:04.939965 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:04.940122 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:04.940160 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:04.940183 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887064940182 us; error 0 us; skew 500 ppm
I20250903 08:11:04.941073 12408 webserver.cc:480] Webserver started at http://127.12.30.1:37255/ using document root <none> and password file <none>
I20250903 08:11:04.941217 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:04.941262 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:04.941332 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:04.941660 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root/instance:
uuid: "09c84ce2093843e1b17fad90b484a397"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:04.942951 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:04.943747 12504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:04.943995 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:04.944056 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
uuid: "09c84ce2093843e1b17fad90b484a397"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:04.944118 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:04.963852 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:04.964222 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:04.964884 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:04.965500 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:04.965543 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:04.965576 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:04.965608 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:04.971302 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.1:38493
I20250903 08:11:04.971355 12567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.1:38493 every 8 connection(s)
I20250903 08:11:04.972139 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:04.973995 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:04.974129 12572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:04.974392 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:04.974491 12570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:04.974691 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:04.974731 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:04.974752 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887064974752 us; error 0 us; skew 500 ppm
I20250903 08:11:04.975493 12408 webserver.cc:480] Webserver started at http://127.12.30.2:39107/ using document root <none> and password file <none>
I20250903 08:11:04.975633 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:04.975683 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:04.975764 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:04.976207 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-1-root/instance:
uuid: "aa688edc69864d758dc75609937cdce6"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:04.978386 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250903 08:11:04.979251 12580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:04.979507 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:04.979573 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-1-root
uuid: "aa688edc69864d758dc75609937cdce6"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:04.979674 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:04.985103 12568 heartbeater.cc:344] Connected to a master server at 127.12.30.62:43647
I20250903 08:11:04.985267 12568 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:04.985577 12568 heartbeater.cc:507] Master 127.12.30.62:43647 requested a full tablet report, sending...
I20250903 08:11:04.987313 12438 ts_manager.cc:194] Registered new tserver with Master: 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:04.988709 12438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56604
I20250903 08:11:04.995374 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:04.995724 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:04.996035 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:04.996529 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:04.996582 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:04.996634 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:04.996688 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:05.003132 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.2:34339
I20250903 08:11:05.003152 12643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.2:34339 every 8 connection(s)
I20250903 08:11:05.003934 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:05.005594 12647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:05.005975 12650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:05.006070 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:05.005978 12648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:05.006417 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:05.006508 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:05.006575 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887065006574 us; error 0 us; skew 500 ppm
I20250903 08:11:05.007787 12408 webserver.cc:480] Webserver started at http://127.12.30.3:43335/ using document root <none> and password file <none>
I20250903 08:11:05.008071 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:05.008162 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:05.008288 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:05.008725 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-2-root/instance:
uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
format_stamp: "Formatted at 2025-09-03 08:11:05 on dist-test-slave-d521"
I20250903 08:11:05.008729 12644 heartbeater.cc:344] Connected to a master server at 127.12.30.62:43647
I20250903 08:11:05.008893 12644 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:05.009097 12644 heartbeater.cc:507] Master 127.12.30.62:43647 requested a full tablet report, sending...
I20250903 08:11:05.009687 12438 ts_manager.cc:194] Registered new tserver with Master: aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339)
I20250903 08:11:05.010387 12438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56614
I20250903 08:11:05.011638 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20250903 08:11:05.012686 12655 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:05.012969 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:05.013038 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-2-root
uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
format_stamp: "Formatted at 2025-09-03 08:11:05 on dist-test-slave-d521"
I20250903 08:11:05.013103 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:05.028059 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:05.028362 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:05.028597 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:05.029038 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:05.029078 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:05.029110 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:05.029134 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:05.034777 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.3:43555
I20250903 08:11:05.034994 12718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.3:43555 every 8 connection(s)
I20250903 08:11:05.038991 12719 heartbeater.cc:344] Connected to a master server at 127.12.30.62:43647
I20250903 08:11:05.039086 12719 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:05.039255 12719 heartbeater.cc:507] Master 127.12.30.62:43647 requested a full tablet report, sending...
I20250903 08:11:05.039824 12438 ts_manager.cc:194] Registered new tserver with Master: 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
I20250903 08:11:05.039937 12408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004713961s
I20250903 08:11:05.040455 12438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56630
I20250903 08:11:05.053954 12408 test_util.cc:276] Using random seed: 627854079
I20250903 08:11:05.062268 12438 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56652:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:11:05.063414 12438 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:11:05.075421 12608 tablet_service.cc:1468] Processing CreateTablet for tablet 2754c413df4a42fda0972454c5935102 (DEFAULT_TABLE table=test-workload [id=6b2e0dd8c37e4023921a978535ec6940]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:05.075479 12683 tablet_service.cc:1468] Processing CreateTablet for tablet 2754c413df4a42fda0972454c5935102 (DEFAULT_TABLE table=test-workload [id=6b2e0dd8c37e4023921a978535ec6940]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:05.075785 12683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2754c413df4a42fda0972454c5935102. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:05.075788 12608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2754c413df4a42fda0972454c5935102. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:05.075871 12532 tablet_service.cc:1468] Processing CreateTablet for tablet 2754c413df4a42fda0972454c5935102 (DEFAULT_TABLE table=test-workload [id=6b2e0dd8c37e4023921a978535ec6940]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:05.076092 12532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2754c413df4a42fda0972454c5935102. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:05.077766 12740 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6: Bootstrap starting.
I20250903 08:11:05.078286 12741 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa: Bootstrap starting.
I20250903 08:11:05.078989 12740 tablet_bootstrap.cc:654] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:05.079244 12741 tablet_bootstrap.cc:654] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:05.079643 12742 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap starting.
I20250903 08:11:05.080016 12740 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6: No bootstrap required, opened a new log
I20250903 08:11:05.080104 12740 ts_tablet_manager.cc:1397] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:11:05.080291 12741 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa: No bootstrap required, opened a new log
I20250903 08:11:05.080372 12741 ts_tablet_manager.cc:1397] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:11:05.080590 12742 tablet_bootstrap.cc:654] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:05.080546 12740 raft_consensus.cc:357] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.080651 12740 raft_consensus.cc:383] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:05.080751 12740 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa688edc69864d758dc75609937cdce6, State: Initialized, Role: FOLLOWER
I20250903 08:11:05.080884 12740 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.081005 12741 raft_consensus.cc:357] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.081218 12741 raft_consensus.cc:383] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:05.081322 12741 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c7cd5cfe7c40f9a732893da08b69fa, State: Initialized, Role: FOLLOWER
I20250903 08:11:05.081424 12644 heartbeater.cc:499] Master 127.12.30.62:43647 was elected leader, sending a full tablet report...
I20250903 08:11:05.081496 12741 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.081650 12742 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: No bootstrap required, opened a new log
I20250903 08:11:05.081724 12742 ts_tablet_manager.cc:1397] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:11:05.081846 12741 ts_tablet_manager.cc:1428] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250903 08:11:05.082041 12719 heartbeater.cc:499] Master 127.12.30.62:43647 was elected leader, sending a full tablet report...
I20250903 08:11:05.081259 12740 ts_tablet_manager.cc:1428] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:05.082149 12742 raft_consensus.cc:357] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.082222 12742 raft_consensus.cc:383] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:05.082252 12742 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Initialized, Role: FOLLOWER
I20250903 08:11:05.082365 12742 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.082705 12742 ts_tablet_manager.cc:1428] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:05.082712 12568 heartbeater.cc:499] Master 127.12.30.62:43647 was elected leader, sending a full tablet report...
I20250903 08:11:05.142843 12747 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:11:05.142967 12747 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.143688 12747 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.144932 12745 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:11:05.145021 12745 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.145500 12745 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.148448 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397" is_pre_election: true
I20250903 08:11:05.148610 12542 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 0.
I20250903 08:11:05.149235 12693 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" is_pre_election: true
I20250903 08:11:05.149253 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa; no voters:
I20250903 08:11:05.149292 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6" is_pre_election: true
I20250903 08:11:05.149380 12693 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa688edc69864d758dc75609937cdce6 in term 0.
I20250903 08:11:05.149500 12618 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 0.
I20250903 08:11:05.149585 12747 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:11:05.149653 12747 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:05.149744 12747 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:05.149806 12583 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6; no voters:
I20250903 08:11:05.149995 12745 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:11:05.150048 12745 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:05.150097 12745 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:05.150655 12747 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.149516 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397" is_pre_election: true
I20250903 08:11:05.150885 12542 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa688edc69864d758dc75609937cdce6 in term 0.
I20250903 08:11:05.150903 12745 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.151172 12747 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 1 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.151218 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:05.151229 12745 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 1 election: Requested vote from peers 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.151374 12618 raft_consensus.cc:2391] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 17c7cd5cfe7c40f9a732893da08b69fa in current term 1: Already voted for candidate aa688edc69864d758dc75609937cdce6 in this term.
I20250903 08:11:05.151496 12541 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:05.151571 12541 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:05.151559 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:05.151556 12693 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:05.151757 12693 raft_consensus.cc:2391] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aa688edc69864d758dc75609937cdce6 in current term 1: Already voted for candidate 17c7cd5cfe7c40f9a732893da08b69fa in this term.
I20250903 08:11:05.151965 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa; no voters: 09c84ce2093843e1b17fad90b484a397, aa688edc69864d758dc75609937cdce6
I20250903 08:11:05.152105 12747 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250903 08:11:05.152415 12541 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa688edc69864d758dc75609937cdce6 in term 1.
I20250903 08:11:05.152688 12581 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397, aa688edc69864d758dc75609937cdce6; no voters: 17c7cd5cfe7c40f9a732893da08b69fa
I20250903 08:11:05.152853 12745 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:11:05.153049 12745 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 LEADER]: Becoming Leader. State: Replica: aa688edc69864d758dc75609937cdce6, State: Running, Role: LEADER
I20250903 08:11:05.153195 12745 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.155042 12436 catalog_manager.cc:5582] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa688edc69864d758dc75609937cdce6 (127.12.30.2). New cstate: current_term: 1 leader_uuid: "aa688edc69864d758dc75609937cdce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:05.176707 12541 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 1 FOLLOWER]: Refusing update from remote peer aa688edc69864d758dc75609937cdce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:11:05.176707 12693 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 1 FOLLOWER]: Refusing update from remote peer aa688edc69864d758dc75609937cdce6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:11:05.177160 12752 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:11:05.177318 12745 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:11:05.230624 12408 test_util.cc:276] Using random seed: 628030750
I20250903 08:11:05.238545 12408 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20250903 08:11:05.461337 12747 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa688edc69864d758dc75609937cdce6)
I20250903 08:11:05.461478 12747 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:11:05.462356 12747 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.462742 12747 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 2 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.463131 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:05.463135 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:05.463490 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa; no voters: 09c84ce2093843e1b17fad90b484a397, aa688edc69864d758dc75609937cdce6
I20250903 08:11:05.463631 12747 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250903 08:11:05.464085 12692 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 2 FOLLOWER]: Rejecting Update request from peer aa688edc69864d758dc75609937cdce6 for earlier term 1. Current term is 2. Ops: []
I20250903 08:11:05.464424 12752 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20250903 08:11:05.464578 12776 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250903 08:11:05.464614 12776 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250903 08:11:05.475557 12748 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aa688edc69864d758dc75609937cdce6)
I20250903 08:11:05.475633 12748 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:11:05.476459 12748 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.476850 12748 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 2 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
I20250903 08:11:05.480423 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:05.480432 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:05.480625 12692 raft_consensus.cc:2391] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 09c84ce2093843e1b17fad90b484a397 in current term 2: Already voted for candidate 17c7cd5cfe7c40f9a732893da08b69fa in this term.
I20250903 08:11:05.480923 12507 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397; no voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6
I20250903 08:11:05.481081 12748 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250903 08:11:05.483762 12776 raft_consensus.cc:3053] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 08:11:05.483831 12776 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: aa688edc69864d758dc75609937cdce6, State: Running, Role: LEADER
I20250903 08:11:05.483994 12776 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.484210 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:11:05.583065 12747 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:05.583190 12747 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:11:05.584005 12747 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.584411 12747 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 3 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.584795 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:05.584795 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:05.584942 12542 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:11:05.584942 12618 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:11:05.585896 12618 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 3.
I20250903 08:11:05.585896 12542 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 3.
I20250903 08:11:05.586196 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa; no voters:
I20250903 08:11:05.586344 12747 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 3 FOLLOWER]: Leader election won for term 3
I20250903 08:11:05.586496 12747 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 3 LEADER]: Becoming Leader. State: Replica: 17c7cd5cfe7c40f9a732893da08b69fa, State: Running, Role: LEADER
I20250903 08:11:05.586634 12747 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.588224 12437 catalog_manager.cc:5582] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa reported cstate change: term changed from 1 to 3, leader changed from aa688edc69864d758dc75609937cdce6 (127.12.30.2) to 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3). New cstate: current_term: 3 leader_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:05.671121 12618 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 3 FOLLOWER]: Refusing update from remote peer 17c7cd5cfe7c40f9a732893da08b69fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20250903 08:11:05.671521 12747 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250903 08:11:05.700390 12542 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 3 FOLLOWER]: Refusing update from remote peer 17c7cd5cfe7c40f9a732893da08b69fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20250903 08:11:05.700870 12747 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20250903 08:11:05.802145 12748 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 17c7cd5cfe7c40f9a732893da08b69fa)
I20250903 08:11:05.802265 12748 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:11:05.803157 12748 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.803529 12748 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 4 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
I20250903 08:11:05.803903 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:05.803908 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:05.804255 12508 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397; no voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6
I20250903 08:11:05.804430 12748 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250903 08:11:05.825812 12542 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 4 FOLLOWER]: Rejecting Update request from peer 17c7cd5cfe7c40f9a732893da08b69fa for earlier term 3. Current term is 4. Ops: []
I20250903 08:11:05.826154 12790 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250903 08:11:05.902416 12542 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 4 FOLLOWER]: Rejecting Update request from peer 17c7cd5cfe7c40f9a732893da08b69fa for earlier term 3. Current term is 4. Ops: []
I20250903 08:11:05.902838 12789 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250903 08:11:05.905860 12790 raft_consensus.cc:3053] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 3 LEADER]: Stepping down as leader of term 3
I20250903 08:11:05.905920 12790 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 17c7cd5cfe7c40f9a732893da08b69fa, State: Running, Role: LEADER
I20250903 08:11:05.906030 12790 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.906195 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:11:05.930974 12748 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:05.931092 12748 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:11:05.932147 12748 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.932456 12748 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 5 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
I20250903 08:11:05.932809 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 5 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:05.932833 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 5 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:05.932915 12692 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:11:05.933557 12692 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c84ce2093843e1b17fad90b484a397 in term 5.
I20250903 08:11:05.933808 12507 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa; no voters: aa688edc69864d758dc75609937cdce6
I20250903 08:11:05.933990 12748 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 5 FOLLOWER]: Leader election won for term 5
I20250903 08:11:05.934150 12748 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 5 LEADER]: Becoming Leader. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Running, Role: LEADER
I20250903 08:11:05.934294 12748 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.935489 12437 catalog_manager.cc:5582] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 reported cstate change: term changed from 3 to 5, leader changed from 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3) to 09c84ce2093843e1b17fad90b484a397 (127.12.30.1). New cstate: current_term: 5 leader_uuid: "09c84ce2093843e1b17fad90b484a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } health_report { overall_health: HEALTHY } } }
I20250903 08:11:05.969244 12776 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 17c7cd5cfe7c40f9a732893da08b69fa)
I20250903 08:11:05.969359 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:11:05.970276 12776 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:05.970657 12776 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 4 election: Requested vote from peers 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:05.971071 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:05.971083 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:05.971249 12692 raft_consensus.cc:2366] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate aa688edc69864d758dc75609937cdce6 for earlier term 4. Current term is 5.
I20250903 08:11:05.971503 12583 leader_election.cc:400] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 4 election: Vote denied by peer 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555) with higher term. Message: Invalid argument: T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate aa688edc69864d758dc75609937cdce6 for earlier term 4. Current term is 5.
I20250903 08:11:05.971546 12583 leader_election.cc:403] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250903 08:11:05.971674 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:11:05.972393 12776 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555) with higher term. Message: Invalid argument: T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate aa688edc69864d758dc75609937cdce6 for earlier term 4. Current term is 5.
I20250903 08:11:06.024042 12618 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 5 FOLLOWER]: Refusing update from remote peer 09c84ce2093843e1b17fad90b484a397: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20250903 08:11:06.024462 12748 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250903 08:11:06.033427 12692 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Refusing update from remote peer 09c84ce2093843e1b17fad90b484a397: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 5 index: 42. (index mismatch)
I20250903 08:11:06.033783 12748 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20250903 08:11:06.135109 12790 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Starting leader election (detected failure of leader 09c84ce2093843e1b17fad90b484a397)
I20250903 08:11:06.135237 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:11:06.136121 12790 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.136518 12790 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 6 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.136924 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.136937 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:06.137187 12692 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 6 FOLLOWER]: Rejecting Update request from peer 09c84ce2093843e1b17fad90b484a397 for earlier term 5. Current term is 6. Ops: []
I20250903 08:11:06.137346 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa; no voters: 09c84ce2093843e1b17fad90b484a397, aa688edc69864d758dc75609937cdce6
I20250903 08:11:06.137522 12790 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250903 08:11:06.137537 12791 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: INVALID_TERM, Last received: 5.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250903 08:11:06.208961 12776 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 09c84ce2093843e1b17fad90b484a397)
I20250903 08:11:06.209090 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:11:06.209997 12776 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.210402 12776 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 6 election: Requested vote from peers 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.210810 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.210836 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 6 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:06.210973 12692 raft_consensus.cc:2391] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate aa688edc69864d758dc75609937cdce6 in current term 6: Already voted for candidate 17c7cd5cfe7c40f9a732893da08b69fa in this term.
I20250903 08:11:06.211043 12748 raft_consensus.cc:3053] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 5 LEADER]: Stepping down as leader of term 5
I20250903 08:11:06.211190 12748 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Running, Role: LEADER
I20250903 08:11:06.211234 12583 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa688edc69864d758dc75609937cdce6; no voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa
I20250903 08:11:06.211376 12776 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250903 08:11:06.211354 12748 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.211508 12748 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:11:06.278007 12790 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:06.278163 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:11:06.279605 12790 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.280231 12790 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 7 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.280649 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.280622 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 7 candidate_status { last_received { term: 5 index: 42 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:06.280783 12618 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:11:06.280786 12542 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:11:06.281687 12618 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 7.
I20250903 08:11:06.281687 12542 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 7.
I20250903 08:11:06.282039 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa; no voters:
I20250903 08:11:06.282198 12790 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 7 FOLLOWER]: Leader election won for term 7
I20250903 08:11:06.282269 12790 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 7 LEADER]: Becoming Leader. State: Replica: 17c7cd5cfe7c40f9a732893da08b69fa, State: Running, Role: LEADER
I20250903 08:11:06.282408 12790 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 5.42, Last appended by leader: 42, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.283624 12437 catalog_manager.cc:5582] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa reported cstate change: term changed from 5 to 7, leader changed from 09c84ce2093843e1b17fad90b484a397 (127.12.30.1) to 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3). New cstate: current_term: 7 leader_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:06.375133 12542 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 7 FOLLOWER]: Refusing update from remote peer 17c7cd5cfe7c40f9a732893da08b69fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 7 index: 43. (index mismatch)
I20250903 08:11:06.375516 12790 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250903 08:11:06.377532 12618 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 7 FOLLOWER]: Refusing update from remote peer 17c7cd5cfe7c40f9a732893da08b69fa: Log matching property violated. Preceding OpId in replica: term: 5 index: 42. Preceding OpId from leader: term: 7 index: 43. (index mismatch)
I20250903 08:11:06.377857 12790 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250903 08:11:06.479358 12776 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 17c7cd5cfe7c40f9a732893da08b69fa)
I20250903 08:11:06.479468 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 7 FOLLOWER]: Advancing to term 8
I20250903 08:11:06.480311 12776 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.480715 12776 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 8 election: Requested vote from peers 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.480988 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.481002 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:06.481406 12583 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa688edc69864d758dc75609937cdce6; no voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa
I20250903 08:11:06.481575 12776 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250903 08:11:06.486752 12618 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 8 FOLLOWER]: Rejecting Update request from peer 17c7cd5cfe7c40f9a732893da08b69fa for earlier term 7. Current term is 8. Ops: []
I20250903 08:11:06.487085 12789 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 }, Status: INVALID_TERM, Last received: 7.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250903 08:11:06.526854 12790 raft_consensus.cc:3053] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 7 LEADER]: Stepping down as leader of term 7
I20250903 08:11:06.526942 12790 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 17c7cd5cfe7c40f9a732893da08b69fa, State: Running, Role: LEADER
I20250903 08:11:06.527137 12790 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.527308 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 7 FOLLOWER]: Advancing to term 8
I20250903 08:11:06.577693 12748 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 17c7cd5cfe7c40f9a732893da08b69fa)
I20250903 08:11:06.577823 12748 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 7 FOLLOWER]: Advancing to term 8
I20250903 08:11:06.578656 12748 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.579051 12748 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 8 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
I20250903 08:11:06.579454 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:06.579455 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 8 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:06.579613 12618 raft_consensus.cc:2391] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 09c84ce2093843e1b17fad90b484a397 in current term 8: Already voted for candidate aa688edc69864d758dc75609937cdce6 in this term.
I20250903 08:11:06.580374 12692 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c84ce2093843e1b17fad90b484a397 in term 8.
I20250903 08:11:06.580677 12507 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397, 17c7cd5cfe7c40f9a732893da08b69fa; no voters: aa688edc69864d758dc75609937cdce6
I20250903 08:11:06.580853 12748 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 8 FOLLOWER]: Leader election won for term 8
I20250903 08:11:06.580929 12748 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 8 LEADER]: Becoming Leader. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Running, Role: LEADER
I20250903 08:11:06.581100 12748 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.582327 12437 catalog_manager.cc:5582] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 reported cstate change: term changed from 7 to 8, leader changed from 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3) to 09c84ce2093843e1b17fad90b484a397 (127.12.30.1). New cstate: current_term: 8 leader_uuid: "09c84ce2093843e1b17fad90b484a397" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } health_report { overall_health: HEALTHY } } }
I20250903 08:11:06.593407 12776 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:06.593487 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 8 FOLLOWER]: Advancing to term 9
I20250903 08:11:06.594162 12776 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.594468 12776 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 9 election: Requested vote from peers 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.594748 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 9 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:06.594748 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "aa688edc69864d758dc75609937cdce6" candidate_term: 9 candidate_status { last_received { term: 7 index: 43 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.594879 12692 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 8 FOLLOWER]: Advancing to term 9
I20250903 08:11:06.595561 12692 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa688edc69864d758dc75609937cdce6 in term 9.
I20250903 08:11:06.595796 12583 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6; no voters: 09c84ce2093843e1b17fad90b484a397
I20250903 08:11:06.595950 12776 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 FOLLOWER]: Leader election won for term 9
I20250903 08:11:06.596127 12776 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 LEADER]: Becoming Leader. State: Replica: aa688edc69864d758dc75609937cdce6, State: Running, Role: LEADER
I20250903 08:11:06.596252 12776 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 7.43, Last appended by leader: 43, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.602062 12436 catalog_manager.cc:5582] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 reported cstate change: term changed from 8 to 9, leader changed from 09c84ce2093843e1b17fad90b484a397 (127.12.30.1) to aa688edc69864d758dc75609937cdce6 (127.12.30.2). New cstate: current_term: 9 leader_uuid: "aa688edc69864d758dc75609937cdce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:06.690521 12618 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 LEADER]: Rejecting Update request from peer 09c84ce2093843e1b17fad90b484a397 for earlier term 8. Current term is 9. Ops: []
I20250903 08:11:06.690977 12748 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250903 08:11:06.691133 12748 consensus_queue.cc:1230] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Peer aa688edc69864d758dc75609937cdce6 log is divergent from this leader: its last log entry 9.44 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 43
I20250903 08:11:06.697651 12692 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 9 FOLLOWER]: Rejecting Update request from peer 09c84ce2093843e1b17fad90b484a397 for earlier term 8. Current term is 9. Ops: []
I20250903 08:11:06.698051 12748 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: INVALID_TERM, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250903 08:11:06.698416 12542 raft_consensus.cc:3053] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 8 LEADER]: Stepping down as leader of term 8
I20250903 08:11:06.698457 12692 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 9 FOLLOWER]: Refusing update from remote peer aa688edc69864d758dc75609937cdce6: Log matching property violated. Preceding OpId in replica: term: 7 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20250903 08:11:06.698482 12542 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Running, Role: LEADER
I20250903 08:11:06.698613 12542 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 8.44, Last appended by leader: 44, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.698781 12776 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250903 08:11:06.698789 12542 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 8 FOLLOWER]: Advancing to term 9
I20250903 08:11:06.700073 12542 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 9 FOLLOWER]: Refusing update from remote peer aa688edc69864d758dc75609937cdce6: Log matching property violated. Preceding OpId in replica: term: 8 index: 44. Preceding OpId from leader: term: 9 index: 44. (term mismatch)
I20250903 08:11:06.700136 12542 pending_rounds.cc:85] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Aborting all ops after (but not including) 43
I20250903 08:11:06.700188 12542 pending_rounds.cc:107] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Aborting uncommitted NO_OP operation due to leader change: 8.44
I20250903 08:11:06.700225 12542 raft_consensus.cc:2887] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250903 08:11:06.700568 12793 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250903 08:11:06.800995 12790 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 9 FOLLOWER]: Starting leader election (detected failure of leader aa688edc69864d758dc75609937cdce6)
I20250903 08:11:06.801093 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 9 FOLLOWER]: Advancing to term 10
I20250903 08:11:06.801880 12790 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.802219 12790 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 10 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.802553 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:06.802561 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.802953 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa; no voters: 09c84ce2093843e1b17fad90b484a397, aa688edc69864d758dc75609937cdce6
I20250903 08:11:06.803135 12790 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250903 08:11:06.813273 12692 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 10 FOLLOWER]: Rejecting Update request from peer aa688edc69864d758dc75609937cdce6 for earlier term 9. Current term is 10. Ops: []
I20250903 08:11:06.813567 12776 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: INVALID_TERM, Last received: 9.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250903 08:11:06.898099 12692 raft_consensus.cc:1238] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 10 FOLLOWER]: Rejecting Update request from peer aa688edc69864d758dc75609937cdce6 for earlier term 9. Current term is 10. Ops: []
I20250903 08:11:06.898437 12794 consensus_queue.cc:1046] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 }, Status: INVALID_TERM, Last received: 9.44, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250903 08:11:06.916625 12790 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:06.916731 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 10 FOLLOWER]: Advancing to term 11
I20250903 08:11:06.917482 12790 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.917797 12790 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 11 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:06.918054 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 11 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:06.918102 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 11 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:06.918416 12656 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa; no voters: 09c84ce2093843e1b17fad90b484a397, aa688edc69864d758dc75609937cdce6
I20250903 08:11:06.918556 12790 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250903 08:11:06.974691 12776 raft_consensus.cc:3053] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 LEADER]: Stepping down as leader of term 9
I20250903 08:11:06.974798 12776 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: aa688edc69864d758dc75609937cdce6, State: Running, Role: LEADER
I20250903 08:11:06.974923 12776 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.975127 12776 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 9 FOLLOWER]: Advancing to term 10
I20250903 08:11:06.995471 12791 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 9 FOLLOWER]: Starting leader election (detected failure of leader aa688edc69864d758dc75609937cdce6)
I20250903 08:11:06.995560 12791 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 9 FOLLOWER]: Advancing to term 10
I20250903 08:11:06.996372 12791 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:06.996759 12791 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 10 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
I20250903 08:11:06.997140 12692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "17c7cd5cfe7c40f9a732893da08b69fa"
I20250903 08:11:06.997140 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "09c84ce2093843e1b17fad90b484a397" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:06.997309 12692 raft_consensus.cc:2366] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 09c84ce2093843e1b17fad90b484a397 for earlier term 10. Current term is 11.
I20250903 08:11:06.997596 12507 leader_election.cc:400] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 10 election: Vote denied by peer 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555) with higher term. Message: Invalid argument: T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 09c84ce2093843e1b17fad90b484a397 for earlier term 10. Current term is 11.
I20250903 08:11:06.997644 12507 leader_election.cc:403] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20250903 08:11:06.997761 12791 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 10 FOLLOWER]: Advancing to term 11
I20250903 08:11:06.998018 12618 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c84ce2093843e1b17fad90b484a397 in term 10.
I20250903 08:11:06.998474 12791 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555) with higher term. Message: Invalid argument: T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 09c84ce2093843e1b17fad90b484a397 for earlier term 10. Current term is 11.
I20250903 08:11:07.001454 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:07.001634 12408 log_reader.cc:385] T 2754c413df4a42fda0972454c5935102: removed 1 log segments from log reader
I20250903 08:11:07.001669 12408 log.cc:1079] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Deleting log segment in path: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root/wals/2754c413df4a42fda0972454c5935102/wal-000000001 (ops 1-40)
I20250903 08:11:07.002632 12408 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20250903 08:11:07.002743 12408 master.cc:561] Master@127.12.30.62:43647 shutting down...
I20250903 08:11:07.089141 12790 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:07.089257 12790 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 11 FOLLOWER]: Advancing to term 12
I20250903 08:11:07.090111 12790 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.090503 12790 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 12 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493)
I20250903 08:11:07.090936 12542 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 12 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "09c84ce2093843e1b17fad90b484a397"
I20250903 08:11:07.090934 12618 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2754c413df4a42fda0972454c5935102" candidate_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" candidate_term: 12 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "aa688edc69864d758dc75609937cdce6"
I20250903 08:11:07.091061 12618 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 10 FOLLOWER]: Advancing to term 12
I20250903 08:11:07.091066 12542 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 11 FOLLOWER]: Advancing to term 12
I20250903 08:11:07.091768 12618 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 12.
I20250903 08:11:07.091820 12542 raft_consensus.cc:2466] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17c7cd5cfe7c40f9a732893da08b69fa in term 12.
I20250903 08:11:07.092078 12659 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6; no voters:
I20250903 08:11:07.092271 12790 raft_consensus.cc:2802] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 12 FOLLOWER]: Leader election won for term 12
I20250903 08:11:07.092422 12790 raft_consensus.cc:695] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 12 LEADER]: Becoming Leader. State: Replica: 17c7cd5cfe7c40f9a732893da08b69fa, State: Running, Role: LEADER
I20250903 08:11:07.092558 12790 consensus_queue.cc:237] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
W20250903 08:11:07.093863 12719 heartbeater.cc:646] Failed to heartbeat to 127.12.30.62:43647 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20250903 08:11:07.098133 12408 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:11:07.098270 12408 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:07.098330 12408 tablet_replica.cc:331] T 00000000000000000000000000000000 P 45f393be2d8c49bb8297ce6b832fac0f: stopping tablet replica
I20250903 08:11:07.111662 12408 master.cc:583] Master@127.12.30.62:43647 shutdown complete.
I20250903 08:11:07.115954 12408 tablet_server.cc:178] TabletServer@127.12.30.1:0 shutting down...
W20250903 08:11:07.169103 12656 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250903 08:11:07.169844 12656 consensus_peers.cc:489] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa -> Peer 09c84ce2093843e1b17fad90b484a397 (127.12.30.1:38493): Couldn't send request to peer 09c84ce2093843e1b17fad90b484a397. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250903 08:11:07.195317 12618 raft_consensus.cc:1273] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 12 FOLLOWER]: Refusing update from remote peer 17c7cd5cfe7c40f9a732893da08b69fa: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 12 index: 45. (index mismatch)
I20250903 08:11:07.195673 12790 consensus_queue.cc:1035] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20250903 08:11:07.223580 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:07.223762 12408 tablet_replica.cc:331] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: stopping tablet replica
I20250903 08:11:07.223850 12408 raft_consensus.cc:2241] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:07.223943 12408 raft_consensus.cc:2270] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:07.236829 12408 tablet_server.cc:195] TabletServer@127.12.30.1:0 shutdown complete.
I20250903 08:11:07.239050 12408 tablet_server.cc:178] TabletServer@127.12.30.2:0 shutting down...
I20250903 08:11:07.254909 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:07.255072 12408 tablet_replica.cc:331] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6: stopping tablet replica
I20250903 08:11:07.255192 12408 raft_consensus.cc:2241] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 12 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:07.255282 12408 raft_consensus.cc:2270] T 2754c413df4a42fda0972454c5935102 P aa688edc69864d758dc75609937cdce6 [term 12 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:07.267470 12408 tablet_server.cc:195] TabletServer@127.12.30.2:0 shutdown complete.
I20250903 08:11:07.269110 12408 tablet_server.cc:178] TabletServer@127.12.30.3:0 shutting down...
W20250903 08:11:07.280925 12659 consensus_peers.cc:489] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa -> Peer aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339): Couldn't send request to peer aa688edc69864d758dc75609937cdce6. Status: Network error: Client connection negotiation failed: client connection to 127.12.30.2:34339: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250903 08:11:07.286624 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:07.286821 12408 tablet_replica.cc:331] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa: stopping tablet replica
I20250903 08:11:07.286916 12408 raft_consensus.cc:2241] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 12 LEADER]: Raft consensus shutting down.
I20250903 08:11:07.287086 12408 raft_consensus.cc:2270] T 2754c413df4a42fda0972454c5935102 P 17c7cd5cfe7c40f9a732893da08b69fa [term 12 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:07.299439 12408 tablet_server.cc:195] TabletServer@127.12.30.3:0 shutdown complete.
I20250903 08:11:07.300871 12408 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20250903 08:11:07.301108 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:07.302457 12799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.302479 12800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.302538 12802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.302757 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:07.302881 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:07.302912 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:07.302925 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887067302925 us; error 0 us; skew 500 ppm
I20250903 08:11:07.303583 12408 webserver.cc:480] Webserver started at http://127.12.30.1:37255/ using document root <none> and password file <none>
I20250903 08:11:07.303695 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:07.303728 12408 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 08:11:07.304765 12408 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:07.307560 12812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.307811 12408 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250903 08:11:07.307883 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
uuid: "09c84ce2093843e1b17fad90b484a397"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:07.307938 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root/data
Total live blocks: 12
Total live bytes: 10829
Total live bytes (after alignment): 57344
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:07.318037 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:07.318244 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:07.318449 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:07.318961 12819 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250903 08:11:07.320230 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250903 08:11:07.320276 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:07.320309 12408 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250903 08:11:07.321012 12408 ts_tablet_manager.cc:610] Registered 1 tablets
I20250903 08:11:07.321048 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:07.321110 12819 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap starting.
I20250903 08:11:07.324673 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.1:38493
I20250903 08:11:07.325995 12819 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=1 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:11:07.326501 12819 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap complete.
I20250903 08:11:07.326771 12819 ts_tablet_manager.cc:1397] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250903 08:11:07.327471 12819 raft_consensus.cc:357] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.327658 12819 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Initialized, Role: FOLLOWER
I20250903 08:11:07.327752 12877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.1:38493 every 8 connection(s)
I20250903 08:11:07.327857 12819 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.328951 12819 ts_tablet_manager.cc:1428] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20250903 08:11:07.329280 12878 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250903 08:11:07.339118 12408 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20250903 08:11:07.339215 12408 tablet_server.cc:178] TabletServer@127.12.30.1:38493 shutting down...
I20250903 08:11:07.421278 12883 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:07.421403 12883 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 12 FOLLOWER]: Advancing to term 13
I20250903 08:11:07.422322 12883 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.422827 12883 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 13 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
W20250903 08:11:07.423856 12816 leader_election.cc:336] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339): Network error: Client connection negotiation failed: client connection to 127.12.30.2:34339: connect: Connection refused (error 111)
W20250903 08:11:07.423980 12815 leader_election.cc:336] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555): Network error: Client connection negotiation failed: client connection to 127.12.30.3:43555: connect: Connection refused (error 111)
I20250903 08:11:07.424041 12815 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397; no voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6
I20250903 08:11:07.424170 12883 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250903 08:11:07.533111 12883 raft_consensus.cc:491] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:07.533234 12883 raft_consensus.cc:3058] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 13 FOLLOWER]: Advancing to term 14
I20250903 08:11:07.534050 12883 raft_consensus.cc:513] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.534400 12883 leader_election.cc:290] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 14 election: Requested vote from peers aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339), 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555)
W20250903 08:11:07.535470 12816 leader_election.cc:336] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer aa688edc69864d758dc75609937cdce6 (127.12.30.2:34339): Network error: Client connection negotiation failed: client connection to 127.12.30.2:34339: connect: Connection refused (error 111)
W20250903 08:11:07.535544 12815 leader_election.cc:336] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 17c7cd5cfe7c40f9a732893da08b69fa (127.12.30.3:43555): Network error: Client connection negotiation failed: client connection to 127.12.30.3:43555: connect: Connection refused (error 111)
I20250903 08:11:07.535606 12815 leader_election.cc:304] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09c84ce2093843e1b17fad90b484a397; no voters: 17c7cd5cfe7c40f9a732893da08b69fa, aa688edc69864d758dc75609937cdce6
I20250903 08:11:07.535742 12883 raft_consensus.cc:2747] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20250903 08:11:07.578760 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:07.579044 12408 tablet_replica.cc:331] stopping tablet replica
I20250903 08:11:07.579173 12408 raft_consensus.cc:2241] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:07.579288 12408 raft_consensus.cc:2270] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:07.591907 12408 tablet_server.cc:195] TabletServer@127.12.30.1:38493 shutdown complete.
I20250903 08:11:07.594008 12408 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20250903 08:11:07.594233 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:07.595661 12887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.595724 12888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.595847 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:07.595934 12890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.596114 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:07.596151 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:07.596167 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887067596167 us; error 0 us; skew 500 ppm
I20250903 08:11:07.596885 12408 webserver.cc:480] Webserver started at http://127.12.30.1:37255/ using document root <none> and password file <none>
I20250903 08:11:07.597003 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:07.597035 12408 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 08:11:07.597985 12408 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:07.600417 12898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.600623 12408 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250903 08:11:07.600721 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
uuid: "09c84ce2093843e1b17fad90b484a397"
format_stamp: "Formatted at 2025-09-03 08:11:04 on dist-test-slave-d521"
I20250903 08:11:07.600805 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1756887064782267-12408-0/minicluster-data/ts-0-root/data
Total live blocks: 10
Total live bytes: 10401
Total live bytes (after alignment): 49152
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:07.606350 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:07.606596 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:07.606865 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:07.608247 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250903 08:11:07.608289 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:07.608952 12408 ts_tablet_manager.cc:610] Registered 1 tablets
I20250903 08:11:07.608994 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:07.609062 12905 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap starting.
I20250903 08:11:07.612510 12905 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=1 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250903 08:11:07.612815 12905 tablet_bootstrap.cc:492] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Bootstrap complete.
I20250903 08:11:07.612968 12905 ts_tablet_manager.cc:1397] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250903 08:11:07.613281 12905 raft_consensus.cc:357] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.613395 12905 raft_consensus.cc:738] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c84ce2093843e1b17fad90b484a397, State: Initialized, Role: FOLLOWER
I20250903 08:11:07.613533 12905 consensus_queue.cc:260] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa688edc69864d758dc75609937cdce6" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 34339 } } peers { permanent_uuid: "17c7cd5cfe7c40f9a732893da08b69fa" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 43555 } } peers { permanent_uuid: "09c84ce2093843e1b17fad90b484a397" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 38493 } }
I20250903 08:11:07.613734 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.1:38493
I20250903 08:11:07.613754 12963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.1:38493 every 8 connection(s)
I20250903 08:11:07.613862 12905 ts_tablet_manager.cc:1428] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20250903 08:11:07.616186 12965 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250903 08:11:07.620990 12408 ts_tablet_manager.cc:1982] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397: failing tablet
I20250903 08:11:07.621160 12905 tablet_replica.cc:331] stopping tablet replica
I20250903 08:11:07.621260 12905 raft_consensus.cc:2241] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:07.621351 12905 raft_consensus.cc:2270] T 2754c413df4a42fda0972454c5935102 P 09c84ce2093843e1b17fad90b484a397 [term 14 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:07.622612 12408 tablet_server.cc:178] TabletServer@127.12.30.1:38493 shutting down...
I20250903 08:11:07.864826 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:07.877197 12408 tablet_server.cc:195] TabletServer@127.12.30.1:38493 shutdown complete.
[ OK ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (3091 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20250903 08:11:07.883211 12408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.30.62:34051
I20250903 08:11:07.883517 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:07.884873 12970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.884982 12973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.885073 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:07.884990 12971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.885313 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:07.885351 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:07.885371 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887067885370 us; error 0 us; skew 500 ppm
I20250903 08:11:07.886045 12408 webserver.cc:480] Webserver started at http://127.12.30.62:46667/ using document root <none> and password file <none>
I20250903 08:11:07.886197 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:07.886245 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:07.886315 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:07.886684 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/master-0-root/instance:
uuid: "351552a25c8a43d0a65ade4963196ddb"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.888031 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:07.888837 12978 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.889050 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:07.889111 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/master-0-root
uuid: "351552a25c8a43d0a65ade4963196ddb"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.889178 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:07.900611 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:07.900905 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:07.904464 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.62:34051
I20250903 08:11:07.904681 13030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.62:34051 every 8 connection(s)
I20250903 08:11:07.905282 13031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:07.906790 13031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb: Bootstrap starting.
I20250903 08:11:07.907435 13031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:07.908193 13031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb: No bootstrap required, opened a new log
I20250903 08:11:07.908489 13031 raft_consensus.cc:357] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351552a25c8a43d0a65ade4963196ddb" member_type: VOTER }
I20250903 08:11:07.908560 13031 raft_consensus.cc:383] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:07.908582 13031 raft_consensus.cc:738] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351552a25c8a43d0a65ade4963196ddb, State: Initialized, Role: FOLLOWER
I20250903 08:11:07.908710 13031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351552a25c8a43d0a65ade4963196ddb" member_type: VOTER }
I20250903 08:11:07.908809 13031 raft_consensus.cc:397] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:11:07.908847 13031 raft_consensus.cc:491] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:11:07.908883 13031 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:07.909451 13031 raft_consensus.cc:513] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351552a25c8a43d0a65ade4963196ddb" member_type: VOTER }
I20250903 08:11:07.909549 13031 leader_election.cc:304] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 351552a25c8a43d0a65ade4963196ddb; no voters:
I20250903 08:11:07.909674 13031 leader_election.cc:290] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:11:07.909823 13034 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:11:07.909999 13034 raft_consensus.cc:695] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 1 LEADER]: Becoming Leader. State: Replica: 351552a25c8a43d0a65ade4963196ddb, State: Running, Role: LEADER
I20250903 08:11:07.910086 13031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:11:07.910137 13034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351552a25c8a43d0a65ade4963196ddb" member_type: VOTER }
I20250903 08:11:07.910511 13036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 351552a25c8a43d0a65ade4963196ddb. Latest consensus state: current_term: 1 leader_uuid: "351552a25c8a43d0a65ade4963196ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351552a25c8a43d0a65ade4963196ddb" member_type: VOTER } }
I20250903 08:11:07.910596 13036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [sys.catalog]: This master's current role is: LEADER
I20250903 08:11:07.910497 13035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "351552a25c8a43d0a65ade4963196ddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351552a25c8a43d0a65ade4963196ddb" member_type: VOTER } }
I20250903 08:11:07.910746 13035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [sys.catalog]: This master's current role is: LEADER
I20250903 08:11:07.911245 13039 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:11:07.911932 13039 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:11:07.912101 12408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 08:11:07.913542 13039 catalog_manager.cc:1349] Generated new cluster ID: e1a9090be6b84462a0ebdd3b5b692df7
I20250903 08:11:07.913587 13039 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:11:07.924510 13039 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:11:07.925009 13039 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:11:07.936687 13039 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb: Generated new TSK 0
I20250903 08:11:07.936808 13039 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:11:07.944262 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:07.945775 13052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.945894 13055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.945930 13053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.946038 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:07.946174 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:07.946213 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:07.946228 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887067946228 us; error 0 us; skew 500 ppm
I20250903 08:11:07.946981 12408 webserver.cc:480] Webserver started at http://127.12.30.1:35907/ using document root <none> and password file <none>
I20250903 08:11:07.947110 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:07.947153 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:07.947232 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:07.947549 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root/instance:
uuid: "90487c84b53440daa01a5039b2653a72"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.948856 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20250903 08:11:07.949590 13060 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.949810 12408 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20250903 08:11:07.949874 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root
uuid: "90487c84b53440daa01a5039b2653a72"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.949939 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:07.958860 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:07.959095 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:07.959306 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:07.959650 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:07.959704 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.959765 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:07.959795 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.963351 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.1:45485
I20250903 08:11:07.963372 13123 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.1:45485 every 8 connection(s)
I20250903 08:11:07.963961 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:07.965425 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.965478 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.965672 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:07.965593 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.965889 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:07.965934 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:07.965955 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887067965954 us; error 0 us; skew 500 ppm
I20250903 08:11:07.966706 12408 webserver.cc:480] Webserver started at http://127.12.30.2:43785/ using document root <none> and password file <none>
I20250903 08:11:07.966871 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:07.966922 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:07.966996 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:07.967322 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-1-root/instance:
uuid: "e6ce9b384db14226b80d2025e5cc34df"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.967945 13124 heartbeater.cc:344] Connected to a master server at 127.12.30.62:34051
I20250903 08:11:07.968034 13124 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:07.968204 13124 heartbeater.cc:507] Master 127.12.30.62:34051 requested a full tablet report, sending...
I20250903 08:11:07.968791 12995 ts_manager.cc:194] Registered new tserver with Master: 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:07.968959 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250903 08:11:07.969502 12995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51226
I20250903 08:11:07.969832 13136 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.970088 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:07.970145 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-1-root
uuid: "e6ce9b384db14226b80d2025e5cc34df"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.970193 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:07.978137 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:07.978391 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:07.978580 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:07.978956 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:07.978991 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.979022 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:07.979051 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.983168 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.2:45243
I20250903 08:11:07.983212 13199 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.2:45243 every 8 connection(s)
I20250903 08:11:07.983804 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:07.985231 13203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.985306 13204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:07.985384 13206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:07.985507 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:07.985695 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:07.985735 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:07.985751 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887067985751 us; error 0 us; skew 500 ppm
I20250903 08:11:07.986474 12408 webserver.cc:480] Webserver started at http://127.12.30.3:44593/ using document root <none> and password file <none>
I20250903 08:11:07.986595 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:07.986670 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:07.986739 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:07.987131 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-2-root/instance:
uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.987221 13200 heartbeater.cc:344] Connected to a master server at 127.12.30.62:34051
I20250903 08:11:07.987298 13200 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:07.987486 13200 heartbeater.cc:507] Master 127.12.30.62:34051 requested a full tablet report, sending...
I20250903 08:11:07.988049 12995 ts_manager.cc:194] Registered new tserver with Master: e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:07.988569 12995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51234
I20250903 08:11:07.988689 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250903 08:11:07.989578 13211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.989794 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:07.989858 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-2-root
uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:07.989926 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:07.998976 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:07.999269 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:07.999503 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:07.999874 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:07.999910 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:07.999958 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:07.999998 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:08.004226 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.3:41321
I20250903 08:11:08.004274 13274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.3:41321 every 8 connection(s)
I20250903 08:11:08.008002 13275 heartbeater.cc:344] Connected to a master server at 127.12.30.62:34051
I20250903 08:11:08.008081 13275 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:08.008247 13275 heartbeater.cc:507] Master 127.12.30.62:34051 requested a full tablet report, sending...
I20250903 08:11:08.008780 12995 ts_manager.cc:194] Registered new tserver with Master: 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321)
I20250903 08:11:08.009009 12408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004439291s
I20250903 08:11:08.009238 12995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51238
I20250903 08:11:08.017992 12408 test_util.cc:276] Using random seed: 630818119
I20250903 08:11:08.024013 12995 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51266:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:11:08.024399 12995 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:11:08.032387 13088 tablet_service.cc:1468] Processing CreateTablet for tablet c7d95e51d9014fb1b067d05e736f07a1 (DEFAULT_TABLE table=test-workload [id=6cfaddaa360249ecaff65c17b7a199a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:08.032644 13088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d95e51d9014fb1b067d05e736f07a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:08.033697 13239 tablet_service.cc:1468] Processing CreateTablet for tablet c7d95e51d9014fb1b067d05e736f07a1 (DEFAULT_TABLE table=test-workload [id=6cfaddaa360249ecaff65c17b7a199a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:08.033778 13164 tablet_service.cc:1468] Processing CreateTablet for tablet c7d95e51d9014fb1b067d05e736f07a1 (DEFAULT_TABLE table=test-workload [id=6cfaddaa360249ecaff65c17b7a199a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:08.033937 13239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d95e51d9014fb1b067d05e736f07a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:08.034024 13164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7d95e51d9014fb1b067d05e736f07a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:08.035050 13296 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Bootstrap starting.
I20250903 08:11:08.036000 13297 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: Bootstrap starting.
I20250903 08:11:08.036139 13296 tablet_bootstrap.cc:654] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:08.036382 13298 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: Bootstrap starting.
I20250903 08:11:08.036979 13297 tablet_bootstrap.cc:654] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:08.037353 13298 tablet_bootstrap.cc:654] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:08.037467 13296 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: No bootstrap required, opened a new log
I20250903 08:11:08.037825 13296 ts_tablet_manager.cc:1397] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250903 08:11:08.037984 13297 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: No bootstrap required, opened a new log
I20250903 08:11:08.038076 13297 ts_tablet_manager.cc:1397] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:11:08.038249 13296 raft_consensus.cc:357] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.038334 13296 raft_consensus.cc:383] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:08.038373 13296 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Initialized, Role: FOLLOWER
I20250903 08:11:08.038491 13296 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.038545 13297 raft_consensus.cc:357] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.038666 13297 raft_consensus.cc:383] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:08.038727 13297 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Initialized, Role: FOLLOWER
I20250903 08:11:08.038828 13296 ts_tablet_manager.cc:1428] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:08.038853 13297 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.038913 13124 heartbeater.cc:499] Master 127.12.30.62:34051 was elected leader, sending a full tablet report...
I20250903 08:11:08.039237 13297 ts_tablet_manager.cc:1428] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:08.039399 13200 heartbeater.cc:499] Master 127.12.30.62:34051 was elected leader, sending a full tablet report...
I20250903 08:11:08.041141 13298 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: No bootstrap required, opened a new log
I20250903 08:11:08.041229 13298 ts_tablet_manager.cc:1397] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250903 08:11:08.041720 13298 raft_consensus.cc:357] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.041828 13298 raft_consensus.cc:383] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:08.041877 13298 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Initialized, Role: FOLLOWER
I20250903 08:11:08.042011 13298 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.042392 13275 heartbeater.cc:499] Master 127.12.30.62:34051 was elected leader, sending a full tablet report...
I20250903 08:11:08.042434 13298 ts_tablet_manager.cc:1428] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:08.069178 13304 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:11:08.069278 13304 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.069631 13304 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.073248 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df" is_pre_election: true
I20250903 08:11:08.073248 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72" is_pre_election: true
I20250903 08:11:08.073424 13174 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 0.
I20250903 08:11:08.073424 13098 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 0.
I20250903 08:11:08.073740 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72; no voters:
I20250903 08:11:08.073907 13304 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:11:08.073966 13304 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:08.074018 13304 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:08.074793 13304 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.075088 13304 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 1 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.075361 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:08.075397 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:08.075466 13174 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:08.075466 13098 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:08.076431 13098 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 1.
I20250903 08:11:08.076431 13174 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 1.
I20250903 08:11:08.076731 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df; no voters:
I20250903 08:11:08.076889 13304 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:11:08.077097 13304 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:08.077212 13304 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.078506 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3). New cstate: current_term: 1 leader_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:08.096478 13174 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 1 FOLLOWER]: Refusing update from remote peer 3599a09f95cd4e08aafaa4bbe02be4d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:11:08.096531 13098 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 1 FOLLOWER]: Refusing update from remote peer 3599a09f95cd4e08aafaa4bbe02be4d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:11:08.096897 13306 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:11:08.097018 13304 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:11:08.104254 13315 mvcc.cc:204] Tried to move back new op lower bound from 7196209430923177984 to 7196209430845435904. Current Snapshot: MvccSnapshot[applied={T|T < 7196209430920433664}]
I20250903 08:11:08.314364 13302 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3599a09f95cd4e08aafaa4bbe02be4d8)
I20250903 08:11:08.314517 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:11:08.315424 13302 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.315852 13302 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 2 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.319500 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:08.319525 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:08.320009 13063 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90487c84b53440daa01a5039b2653a72; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:08.320101 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Rejecting Update request from peer 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 1. Current term is 2. Ops: []
I20250903 08:11:08.320199 13302 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250903 08:11:08.320459 13328 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 1.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250903 08:11:08.320715 13306 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 1 LEADER]: Stepping down as leader of term 1
I20250903 08:11:08.320776 13306 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:08.320931 13306 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.321079 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:11:08.336565 13303 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3599a09f95cd4e08aafaa4bbe02be4d8)
I20250903 08:11:08.336642 13303 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 1 FOLLOWER]: Advancing to term 2
I20250903 08:11:08.337356 13303 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.337746 13303 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 2 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:08.340605 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:08.340710 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:08.340817 13098 raft_consensus.cc:2391] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e6ce9b384db14226b80d2025e5cc34df in current term 2: Already voted for candidate 90487c84b53440daa01a5039b2653a72 in this term.
I20250903 08:11:08.341485 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ce9b384db14226b80d2025e5cc34df in term 2.
I20250903 08:11:08.341717 13139 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df; no voters: 90487c84b53440daa01a5039b2653a72
I20250903 08:11:08.341858 13303 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 2 FOLLOWER]: Leader election won for term 2
I20250903 08:11:08.341986 13303 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 2 LEADER]: Becoming Leader. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:08.342118 13303 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.343324 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df reported cstate change: term changed from 1 to 2, leader changed from 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3) to e6ce9b384db14226b80d2025e5cc34df (127.12.30.2). New cstate: current_term: 2 leader_uuid: "e6ce9b384db14226b80d2025e5cc34df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: HEALTHY } } }
I20250903 08:11:08.443624 13098 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Refusing update from remote peer e6ce9b384db14226b80d2025e5cc34df: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250903 08:11:08.444013 13303 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Connected to new peer: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250903 08:11:08.461284 13249 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 2 FOLLOWER]: Refusing update from remote peer e6ce9b384db14226b80d2025e5cc34df: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20250903 08:11:08.461627 13303 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250903 08:11:08.562729 13306 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6ce9b384db14226b80d2025e5cc34df)
I20250903 08:11:08.562821 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:11:08.563560 13306 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.563879 13306 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 3 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.564203 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:08.564203 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:08.564572 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8; no voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:08.564713 13306 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250903 08:11:08.578819 13249 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Rejecting Update request from peer e6ce9b384db14226b80d2025e5cc34df for earlier term 2. Current term is 3. Ops: []
I20250903 08:11:08.579105 13336 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: INVALID_TERM, Last received: 2.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250903 08:11:08.646083 13302 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6ce9b384db14226b80d2025e5cc34df)
I20250903 08:11:08.646159 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:11:08.647006 13302 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.647336 13302 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 3 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.647642 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:08.647652 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:08.647868 13249 raft_consensus.cc:2391] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 90487c84b53440daa01a5039b2653a72 in current term 3: Already voted for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in this term.
I20250903 08:11:08.648113 13063 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90487c84b53440daa01a5039b2653a72; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:08.648265 13302 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250903 08:11:08.649281 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 3 FOLLOWER]: Rejecting Update request from peer e6ce9b384db14226b80d2025e5cc34df for earlier term 2. Current term is 3. Ops: []
I20250903 08:11:08.649540 13336 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 2.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250903 08:11:08.672166 13249 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Rejecting Update request from peer e6ce9b384db14226b80d2025e5cc34df for earlier term 2. Current term is 3. Ops: []
I20250903 08:11:08.672480 13336 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: INVALID_TERM, Last received: 2.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250903 08:11:08.688177 13306 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:08.688251 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:11:08.689080 13306 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.689370 13306 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 4 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.689640 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:08.689776 13098 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:11:08.689698 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:08.690488 13098 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 4.
I20250903 08:11:08.690742 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72; no voters: e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:08.690922 13306 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 4 FOLLOWER]: Leader election won for term 4
I20250903 08:11:08.691010 13306 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 4 LEADER]: Becoming Leader. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:08.691133 13306 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.692268 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 reported cstate change: term changed from 2 to 4, leader changed from e6ce9b384db14226b80d2025e5cc34df (127.12.30.2) to 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3). New cstate: current_term: 4 leader_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:08.702476 13303 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 2 LEADER]: Stepping down as leader of term 2
I20250903 08:11:08.702536 13303 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:08.702654 13303 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.702777 13303 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 2 FOLLOWER]: Advancing to term 3
I20250903 08:11:08.786590 13174 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 3 FOLLOWER]: Advancing to term 4
I20250903 08:11:08.787567 13174 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 4 FOLLOWER]: Refusing update from remote peer 3599a09f95cd4e08aafaa4bbe02be4d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20250903 08:11:08.787971 13306 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250903 08:11:08.814513 13098 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 4 FOLLOWER]: Refusing update from remote peer 3599a09f95cd4e08aafaa4bbe02be4d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20250903 08:11:08.814851 13306 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20250903 08:11:08.916316 13302 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3599a09f95cd4e08aafaa4bbe02be4d8)
I20250903 08:11:08.916411 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:11:08.917238 13302 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:08.917550 13302 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 5 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:08.917856 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:08.917891 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:08.918279 13061 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90487c84b53440daa01a5039b2653a72; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:08.918443 13302 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250903 08:11:08.918800 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 5 FOLLOWER]: Rejecting Update request from peer 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 4. Current term is 5. Ops: []
I20250903 08:11:08.919086 13337 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 4.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250903 08:11:09.018045 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 5 FOLLOWER]: Rejecting Update request from peer 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 4. Current term is 5. Ops: []
I20250903 08:11:09.018415 13337 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 4.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250903 08:11:09.062284 13306 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 4 LEADER]: Stepping down as leader of term 4
I20250903 08:11:09.062379 13306 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:09.062543 13306 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.062724 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:11:09.063999 13302 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:09.064078 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:11:09.064774 13302 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.065089 13302 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 6 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:09.065454 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:09.065454 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 6 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:09.065586 13249 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:11:09.066332 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90487c84b53440daa01a5039b2653a72 in term 6.
I20250903 08:11:09.066567 13063 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72; no voters: e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:09.066782 13302 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 6 FOLLOWER]: Leader election won for term 6
I20250903 08:11:09.066946 13302 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 6 LEADER]: Becoming Leader. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:09.067085 13302 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.068392 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 reported cstate change: term changed from 4 to 6, leader changed from 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3) to 90487c84b53440daa01a5039b2653a72 (127.12.30.1). New cstate: current_term: 6 leader_uuid: "90487c84b53440daa01a5039b2653a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:09.097437 13303 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3599a09f95cd4e08aafaa4bbe02be4d8)
I20250903 08:11:09.097524 13303 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 4 FOLLOWER]: Advancing to term 5
I20250903 08:11:09.098253 13303 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.098598 13303 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 5 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:09.098891 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:09.098946 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:09.098997 13249 raft_consensus.cc:2366] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6ce9b384db14226b80d2025e5cc34df for earlier term 5. Current term is 6.
I20250903 08:11:09.099341 13139 leader_election.cc:400] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 5 election: Vote denied by peer 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321) with higher term. Message: Invalid argument: T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6ce9b384db14226b80d2025e5cc34df for earlier term 5. Current term is 6.
I20250903 08:11:09.099388 13139 leader_election.cc:403] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250903 08:11:09.099483 13303 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 5 FOLLOWER]: Advancing to term 6
I20250903 08:11:09.100144 13303 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321) with higher term. Message: Invalid argument: T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6ce9b384db14226b80d2025e5cc34df for earlier term 5. Current term is 6.
I20250903 08:11:09.146203 13249 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Refusing update from remote peer 90487c84b53440daa01a5039b2653a72: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
I20250903 08:11:09.146561 13302 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250903 08:11:09.152768 13174 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 6 FOLLOWER]: Refusing update from remote peer 90487c84b53440daa01a5039b2653a72: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 6 index: 18. (index mismatch)
I20250903 08:11:09.153137 13302 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20250903 08:11:09.347561 13303 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 6 FOLLOWER]: Starting leader election (detected failure of leader 90487c84b53440daa01a5039b2653a72)
I20250903 08:11:09.347679 13303 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:11:09.348505 13303 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.348881 13303 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 7 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:09.349182 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 7 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:09.349192 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 7 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:09.349543 13137 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6ce9b384db14226b80d2025e5cc34df; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72
I20250903 08:11:09.349676 13303 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250903 08:11:09.356518 13306 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 90487c84b53440daa01a5039b2653a72)
I20250903 08:11:09.356595 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:11:09.357291 13174 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 7 FOLLOWER]: Rejecting Update request from peer 90487c84b53440daa01a5039b2653a72 for earlier term 6. Current term is 7. Ops: []
I20250903 08:11:09.357460 13306 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.357646 13302 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: INVALID_TERM, Last received: 6.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250903 08:11:09.357805 13306 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 7 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:09.358072 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:09.358079 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:09.358201 13174 raft_consensus.cc:2391] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in current term 7: Already voted for candidate e6ce9b384db14226b80d2025e5cc34df in this term.
I20250903 08:11:09.358402 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8; no voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:09.358561 13306 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250903 08:11:09.366986 13249 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 7 FOLLOWER]: Rejecting Update request from peer 90487c84b53440daa01a5039b2653a72 for earlier term 6. Current term is 7. Ops: []
I20250903 08:11:09.367290 13339 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: INVALID_TERM, Last received: 6.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250903 08:11:09.386962 13302 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 6 LEADER]: Stepping down as leader of term 6
I20250903 08:11:09.387033 13302 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:09.387173 13302 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 6.18, Last appended by leader: 18, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.387320 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 6 FOLLOWER]: Advancing to term 7
I20250903 08:11:09.460194 13303 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:09.460301 13303 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 7 FOLLOWER]: Advancing to term 8
I20250903 08:11:09.461149 13303 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.461488 13303 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 8 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:09.461901 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 8 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:09.461915 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 8 candidate_status { last_received { term: 6 index: 18 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:09.462006 13249 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 7 FOLLOWER]: Advancing to term 8
I20250903 08:11:09.462054 13098 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 7 FOLLOWER]: Advancing to term 8
I20250903 08:11:09.462698 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ce9b384db14226b80d2025e5cc34df in term 8.
I20250903 08:11:09.462795 13098 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ce9b384db14226b80d2025e5cc34df in term 8.
I20250903 08:11:09.462972 13139 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df; no voters:
I20250903 08:11:09.463132 13303 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 8 FOLLOWER]: Leader election won for term 8
I20250903 08:11:09.463286 13303 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 8 LEADER]: Becoming Leader. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:09.463433 13303 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 6.18, Last appended by leader: 18, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.464718 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df reported cstate change: term changed from 6 to 8, leader changed from 90487c84b53440daa01a5039b2653a72 (127.12.30.1) to e6ce9b384db14226b80d2025e5cc34df (127.12.30.2). New cstate: current_term: 8 leader_uuid: "e6ce9b384db14226b80d2025e5cc34df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: HEALTHY } } }
I20250903 08:11:09.559098 13098 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 8 FOLLOWER]: Refusing update from remote peer e6ce9b384db14226b80d2025e5cc34df: Log matching property violated. Preceding OpId in replica: term: 6 index: 18. Preceding OpId from leader: term: 8 index: 19. (index mismatch)
I20250903 08:11:09.559514 13303 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Connected to new peer: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250903 08:11:09.563513 13249 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 8 FOLLOWER]: Refusing update from remote peer e6ce9b384db14226b80d2025e5cc34df: Log matching property violated. Preceding OpId in replica: term: 6 index: 18. Preceding OpId from leader: term: 8 index: 19. (index mismatch)
I20250903 08:11:09.563827 13303 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20250903 08:11:09.751945 13302 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6ce9b384db14226b80d2025e5cc34df)
I20250903 08:11:09.752043 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 8 FOLLOWER]: Advancing to term 9
I20250903 08:11:09.752902 13302 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.753239 13302 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 9 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:09.753515 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:09.753574 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:09.753890 13061 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90487c84b53440daa01a5039b2653a72; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:09.754045 13302 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250903 08:11:09.760775 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 9 FOLLOWER]: Rejecting Update request from peer e6ce9b384db14226b80d2025e5cc34df for earlier term 8. Current term is 9. Ops: []
I20250903 08:11:09.761118 13340 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 8.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250903 08:11:09.797360 13341 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 8 LEADER]: Stepping down as leader of term 8
I20250903 08:11:09.797428 13341 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:09.797592 13341 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 8.19, Last appended by leader: 19, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.797755 13341 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 8 FOLLOWER]: Advancing to term 9
I20250903 08:11:09.838027 13306 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6ce9b384db14226b80d2025e5cc34df)
I20250903 08:11:09.838116 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 8 FOLLOWER]: Advancing to term 9
I20250903 08:11:09.838949 13306 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.839308 13306 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 9 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:09.839728 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:09.839740 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:09.839900 13098 raft_consensus.cc:2391] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in current term 9: Already voted for candidate 90487c84b53440daa01a5039b2653a72 in this term.
I20250903 08:11:09.840570 13174 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 9.
I20250903 08:11:09.840865 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df; no voters: 90487c84b53440daa01a5039b2653a72
I20250903 08:11:09.841043 13306 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 9 FOLLOWER]: Leader election won for term 9
I20250903 08:11:09.841217 13306 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 9 LEADER]: Becoming Leader. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:09.841352 13306 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 8.19, Last appended by leader: 19, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.842641 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 reported cstate change: term changed from 8 to 9, leader changed from e6ce9b384db14226b80d2025e5cc34df (127.12.30.2) to 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3). New cstate: current_term: 9 leader_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:09.893461 13302 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:09.893576 13302 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 9 FOLLOWER]: Advancing to term 10
I20250903 08:11:09.894380 13302 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.894742 13302 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 10 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:09.895112 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:09.895119 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 10 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:09.895221 13174 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 9 FOLLOWER]: Advancing to term 10
I20250903 08:11:09.895903 13174 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90487c84b53440daa01a5039b2653a72 in term 10.
I20250903 08:11:09.896193 13061 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8
I20250903 08:11:09.896377 13302 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 FOLLOWER]: Leader election won for term 10
I20250903 08:11:09.896540 13302 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 LEADER]: Becoming Leader. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:09.896703 13302 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 8.19, Last appended by leader: 19, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.921178 13174 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 10 FOLLOWER]: Rejecting Update request from peer 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 9. Current term is 10. Ops: []
I20250903 08:11:09.921511 13306 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250903 08:11:09.928617 12993 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 reported cstate change: term changed from 9 to 10, leader changed from 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3) to 90487c84b53440daa01a5039b2653a72 (127.12.30.1). New cstate: current_term: 10 leader_uuid: "90487c84b53440daa01a5039b2653a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:09.939064 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 LEADER]: Rejecting Update request from peer 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 9. Current term is 10. Ops: []
I20250903 08:11:09.939415 13342 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250903 08:11:09.939509 13342 consensus_queue.cc:1230] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer 90487c84b53440daa01a5039b2653a72 log is divergent from this leader: its last log entry 10.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20250903 08:11:09.980211 13174 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 10 FOLLOWER]: Refusing update from remote peer 90487c84b53440daa01a5039b2653a72: Log matching property violated. Preceding OpId in replica: term: 8 index: 19. Preceding OpId from leader: term: 10 index: 20. (index mismatch)
I20250903 08:11:09.980589 13302 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250903 08:11:09.985716 13306 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 9 LEADER]: Stepping down as leader of term 9
I20250903 08:11:09.985781 13306 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:09.985893 13306 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 9.20, Last appended by leader: 20, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:09.986024 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 9 FOLLOWER]: Advancing to term 10
I20250903 08:11:09.998638 13249 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 10 FOLLOWER]: Refusing update from remote peer 90487c84b53440daa01a5039b2653a72: Log matching property violated. Preceding OpId in replica: term: 9 index: 20. Preceding OpId from leader: term: 10 index: 20. (term mismatch)
I20250903 08:11:09.998706 13249 pending_rounds.cc:85] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: Aborting all ops after (but not including) 19
I20250903 08:11:09.998724 13249 pending_rounds.cc:107] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: Aborting uncommitted NO_OP operation due to leader change: 9.20
I20250903 08:11:09.998741 13249 raft_consensus.cc:2887] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250903 08:11:09.999060 13302 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20250903 08:11:10.083220 13341 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 10 FOLLOWER]: Starting leader election (detected failure of leader 90487c84b53440daa01a5039b2653a72)
I20250903 08:11:10.083331 13341 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 10 FOLLOWER]: Advancing to term 11
I20250903 08:11:10.084141 13341 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.084504 13341 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 11 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:10.084930 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.084932 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.085323 13137 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6ce9b384db14226b80d2025e5cc34df; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72
I20250903 08:11:10.085477 13341 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250903 08:11:10.104403 13174 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 11 FOLLOWER]: Rejecting Update request from peer 90487c84b53440daa01a5039b2653a72 for earlier term 10. Current term is 11. Ops: []
I20250903 08:11:10.104744 13345 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: INVALID_TERM, Last received: 10.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250903 08:11:10.192790 13341 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:10.192921 13341 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 11 FOLLOWER]: Advancing to term 12
I20250903 08:11:10.193687 13341 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.193885 13174 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Rejecting Update request from peer 90487c84b53440daa01a5039b2653a72 for earlier term 10. Current term is 12. Ops: []
I20250903 08:11:10.194057 13341 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 12 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:10.194314 13302 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: INVALID_TERM, Last received: 10.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250903 08:11:10.194340 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 12 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.194340 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 12 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.194758 13137 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6ce9b384db14226b80d2025e5cc34df; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72
I20250903 08:11:10.194907 13341 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250903 08:11:10.254032 13345 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 LEADER]: Stepping down as leader of term 10
I20250903 08:11:10.254132 13345 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:10.254297 13345 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 10.20, Last appended by leader: 20, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.254480 13345 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 10 FOLLOWER]: Advancing to term 11
I20250903 08:11:10.260196 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.261436 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.263706 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.267884 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.276213 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.292495 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.310554 13306 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 90487c84b53440daa01a5039b2653a72)
I20250903 08:11:10.310647 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 10 FOLLOWER]: Advancing to term 11
I20250903 08:11:10.311563 13306 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.311944 13306 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 11 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:10.312364 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.312371 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:10.312521 13174 raft_consensus.cc:2366] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 11. Current term is 12.
I20250903 08:11:10.312804 13212 leader_election.cc:400] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 11 election: Vote denied by peer e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243) with higher term. Message: Invalid argument: T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 11. Current term is 12.
I20250903 08:11:10.312844 13212 leader_election.cc:403] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20250903 08:11:10.312949 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 11 FOLLOWER]: Advancing to term 12
I20250903 08:11:10.313354 13098 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 11.
I20250903 08:11:10.313760 13306 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243) with higher term. Message: Invalid argument: T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 11. Current term is 12.
I20250903 08:11:10.324833 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.389061 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.398597 13341 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:10.398725 13341 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 12 FOLLOWER]: Advancing to term 13
I20250903 08:11:10.399569 13341 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.399986 13341 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 13 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:10.400369 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 13 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.400367 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 13 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.400491 13249 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 12 FOLLOWER]: Advancing to term 13
I20250903 08:11:10.400547 13098 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 11 FOLLOWER]: Advancing to term 13
I20250903 08:11:10.401268 13098 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ce9b384db14226b80d2025e5cc34df in term 13.
I20250903 08:11:10.401276 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ce9b384db14226b80d2025e5cc34df in term 13.
I20250903 08:11:10.401515 13137 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df; no voters:
I20250903 08:11:10.401697 13341 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 13 FOLLOWER]: Leader election won for term 13
I20250903 08:11:10.401827 13341 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 13 LEADER]: Becoming Leader. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:10.401948 13341 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 10.20, Last appended by leader: 20, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.403100 12993 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df reported cstate change: term changed from 10 to 13, leader changed from 90487c84b53440daa01a5039b2653a72 (127.12.30.1) to e6ce9b384db14226b80d2025e5cc34df (127.12.30.2). New cstate: current_term: 13 leader_uuid: "e6ce9b384db14226b80d2025e5cc34df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: HEALTHY } } }
I20250903 08:11:10.507045 13345 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:10.507160 13345 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 13 FOLLOWER]: Advancing to term 14
I20250903 08:11:10.507989 13345 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.508351 13345 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 14 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:10.508848 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 14 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.508848 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 14 candidate_status { last_received { term: 10 index: 20 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:10.509028 13249 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 13 FOLLOWER]: Advancing to term 14
I20250903 08:11:10.509824 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90487c84b53440daa01a5039b2653a72 in term 14.
I20250903 08:11:10.510073 13063 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72; no voters: e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:10.510244 13345 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 FOLLOWER]: Leader election won for term 14
I20250903 08:11:10.510368 13345 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 LEADER]: Becoming Leader. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:10.510504 13345 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 10.20, Last appended by leader: 20, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.511804 12993 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 reported cstate change: term changed from 13 to 14, leader changed from e6ce9b384db14226b80d2025e5cc34df (127.12.30.2) to 90487c84b53440daa01a5039b2653a72 (127.12.30.1). New cstate: current_term: 14 leader_uuid: "90487c84b53440daa01a5039b2653a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:10.517325 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.523357 13098 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 LEADER]: Rejecting Update request from peer e6ce9b384db14226b80d2025e5cc34df for earlier term 13. Current term is 14. Ops: []
I20250903 08:11:10.523690 13341 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250903 08:11:10.523800 13341 consensus_queue.cc:1230] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer 90487c84b53440daa01a5039b2653a72 log is divergent from this leader: its last log entry 14.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20250903 08:11:10.523859 13346 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 13 LEADER]: Stepping down as leader of term 13
I20250903 08:11:10.523927 13346 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:10.524060 13346 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 13.21, Last appended by leader: 21, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.524184 13346 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 13 FOLLOWER]: Advancing to term 14
I20250903 08:11:10.586972 13174 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 14 FOLLOWER]: Refusing update from remote peer 90487c84b53440daa01a5039b2653a72: Log matching property violated. Preceding OpId in replica: term: 13 index: 21. Preceding OpId from leader: term: 14 index: 21. (term mismatch)
I20250903 08:11:10.587059 13174 pending_rounds.cc:85] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: Aborting all ops after (but not including) 20
I20250903 08:11:10.587076 13174 pending_rounds.cc:107] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: Aborting uncommitted NO_OP operation due to leader change: 13.21
I20250903 08:11:10.587095 13174 raft_consensus.cc:2887] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250903 08:11:10.587491 13345 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250903 08:11:10.589524 13249 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 14 FOLLOWER]: Refusing update from remote peer 90487c84b53440daa01a5039b2653a72: Log matching property violated. Preceding OpId in replica: term: 10 index: 20. Preceding OpId from leader: term: 14 index: 21. (index mismatch)
I20250903 08:11:10.589995 13345 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20250903 08:11:10.691952 13306 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 90487c84b53440daa01a5039b2653a72)
I20250903 08:11:10.692082 13306 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 14 FOLLOWER]: Advancing to term 15
I20250903 08:11:10.693066 13306 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.693418 13306 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 15 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:10.693538 13249 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 FOLLOWER]: Rejecting Update request from peer 90487c84b53440daa01a5039b2653a72 for earlier term 14. Current term is 15. Ops: []
I20250903 08:11:10.693658 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 21 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.694093 13343 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: INVALID_TERM, Last received: 14.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250903 08:11:10.694134 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 21 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:10.694226 13343 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 LEADER]: Stepping down as leader of term 14
I20250903 08:11:10.694234 13174 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 14 FOLLOWER]: Advancing to term 15
I20250903 08:11:10.694311 13343 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:10.694470 13343 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 14.21, Last appended by leader: 21, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.694617 13343 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 14 FOLLOWER]: Advancing to term 15
I20250903 08:11:10.695188 13174 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in term 15.
I20250903 08:11:10.695449 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df; no voters: 90487c84b53440daa01a5039b2653a72
I20250903 08:11:10.695598 13306 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 FOLLOWER]: Leader election won for term 15
I20250903 08:11:10.695737 13306 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 LEADER]: Becoming Leader. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:10.695855 13306 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 14.21, Last appended by leader: 21, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.696964 12993 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 reported cstate change: term changed from 14 to 15, leader changed from 90487c84b53440daa01a5039b2653a72 (127.12.30.1) to 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3). New cstate: current_term: 15 leader_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:10.773641 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:10.781306 13174 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 15 FOLLOWER]: Refusing update from remote peer 3599a09f95cd4e08aafaa4bbe02be4d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 21. Preceding OpId from leader: term: 15 index: 22. (index mismatch)
I20250903 08:11:10.781694 13306 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250903 08:11:10.783926 13098 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 15 FOLLOWER]: Refusing update from remote peer 3599a09f95cd4e08aafaa4bbe02be4d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 21. Preceding OpId from leader: term: 15 index: 22. (index mismatch)
I20250903 08:11:10.784264 13351 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20250903 08:11:10.884083 13346 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3599a09f95cd4e08aafaa4bbe02be4d8)
I20250903 08:11:10.884220 13346 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 15 FOLLOWER]: Advancing to term 16
I20250903 08:11:10.885258 13346 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.885578 13346 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 16 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:10.885834 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 16 candidate_status { last_received { term: 15 index: 22 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.885876 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 16 candidate_status { last_received { term: 15 index: 22 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.886197 13137 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6ce9b384db14226b80d2025e5cc34df; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72
I20250903 08:11:10.886340 13346 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250903 08:11:10.893411 13174 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 16 FOLLOWER]: Rejecting Update request from peer 3599a09f95cd4e08aafaa4bbe02be4d8 for earlier term 15. Current term is 16. Ops: []
I20250903 08:11:10.893759 13351 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 }, Status: INVALID_TERM, Last received: 15.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250903 08:11:10.893921 13350 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 LEADER]: Stepping down as leader of term 15
I20250903 08:11:10.893994 13350 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 3599a09f95cd4e08aafaa4bbe02be4d8, State: Running, Role: LEADER
I20250903 08:11:10.894133 13350 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 15.22, Last appended by leader: 22, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.894294 13350 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 15 FOLLOWER]: Advancing to term 16
I20250903 08:11:10.984292 13343 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 3599a09f95cd4e08aafaa4bbe02be4d8)
I20250903 08:11:10.984421 13343 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 15 FOLLOWER]: Advancing to term 16
I20250903 08:11:10.985327 13343 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.985743 13343 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 16 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:10.986162 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 16 candidate_status { last_received { term: 15 index: 22 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:10.986162 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 16 candidate_status { last_received { term: 15 index: 22 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.986337 13174 raft_consensus.cc:2391] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 90487c84b53440daa01a5039b2653a72 in current term 16: Already voted for candidate e6ce9b384db14226b80d2025e5cc34df in this term.
I20250903 08:11:10.987084 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90487c84b53440daa01a5039b2653a72 in term 16.
I20250903 08:11:10.987308 13063 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, 90487c84b53440daa01a5039b2653a72; no voters: e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:10.987462 13343 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 16 FOLLOWER]: Leader election won for term 16
I20250903 08:11:10.987565 13343 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 16 LEADER]: Becoming Leader. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:10.987717 13343 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 15.22, Last appended by leader: 22, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.989037 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 reported cstate change: term changed from 15 to 16, leader changed from 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3) to 90487c84b53440daa01a5039b2653a72 (127.12.30.1). New cstate: current_term: 16 leader_uuid: "90487c84b53440daa01a5039b2653a72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:10.992758 13346 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:10.992823 13346 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 16 FOLLOWER]: Advancing to term 17
I20250903 08:11:10.993445 13346 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.993702 13346 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 17 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485)
I20250903 08:11:10.993980 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 17 candidate_status { last_received { term: 15 index: 22 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:10.993988 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "e6ce9b384db14226b80d2025e5cc34df" candidate_term: 17 candidate_status { last_received { term: 15 index: 22 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:10.994100 13249 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 16 FOLLOWER]: Advancing to term 17
I20250903 08:11:10.994697 13249 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ce9b384db14226b80d2025e5cc34df in term 17.
I20250903 08:11:10.994932 13139 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df; no voters: 90487c84b53440daa01a5039b2653a72
I20250903 08:11:10.995095 13346 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 17 FOLLOWER]: Leader election won for term 17
I20250903 08:11:10.995172 13346 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 17 LEADER]: Becoming Leader. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:10.995290 13346 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 15.22, Last appended by leader: 22, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:10.996330 12995 catalog_manager.cc:5582] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df reported cstate change: term changed from 16 to 17, leader changed from 90487c84b53440daa01a5039b2653a72 (127.12.30.1) to e6ce9b384db14226b80d2025e5cc34df (127.12.30.2). New cstate: current_term: 17 leader_uuid: "e6ce9b384db14226b80d2025e5cc34df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } health_report { overall_health: HEALTHY } } }
I20250903 08:11:11.079341 13249 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 17 FOLLOWER]: Refusing update from remote peer e6ce9b384db14226b80d2025e5cc34df: Log matching property violated. Preceding OpId in replica: term: 15 index: 22. Preceding OpId from leader: term: 17 index: 23. (index mismatch)
I20250903 08:11:11.079758 13346 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250903 08:11:11.082031 13098 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 16 LEADER]: Stepping down as leader of term 16
I20250903 08:11:11.082088 13098 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:11.082207 13098 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 16.23, Last appended by leader: 23, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.082335 13098 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 16 FOLLOWER]: Advancing to term 17
I20250903 08:11:11.083587 13098 raft_consensus.cc:1273] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 17 FOLLOWER]: Refusing update from remote peer e6ce9b384db14226b80d2025e5cc34df: Log matching property violated. Preceding OpId in replica: term: 16 index: 23. Preceding OpId from leader: term: 17 index: 23. (term mismatch)
I20250903 08:11:11.083662 13098 pending_rounds.cc:85] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Aborting all ops after (but not including) 22
I20250903 08:11:11.083690 13098 pending_rounds.cc:107] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Aborting uncommitted NO_OP operation due to leader change: 16.23
I20250903 08:11:11.083725 13098 raft_consensus.cc:2887] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250903 08:11:11.084066 13341 consensus_queue.cc:1035] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Connected to new peer: Peer: permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20250903 08:11:11.286053 12408 timestamp_advancement-itest.cc:267] GCing logs...
I20250903 08:11:11.286216 12408 log_reader.cc:385] T c7d95e51d9014fb1b067d05e736f07a1: removed 1 log segments from log reader
I20250903 08:11:11.286243 12408 log.cc:1079] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Deleting log segment in path: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root/wals/c7d95e51d9014fb1b067d05e736f07a1/wal-000000001 (ops 1-15)
I20250903 08:11:11.287356 12408 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20250903 08:11:11.287467 12408 master.cc:561] Master@127.12.30.62:34051 shutting down...
I20250903 08:11:11.357602 13350 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6ce9b384db14226b80d2025e5cc34df)
I20250903 08:11:11.357729 13350 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 17 FOLLOWER]: Advancing to term 18
I20250903 08:11:11.358561 13350 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.358920 13350 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 18 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:11.359256 13098 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 23 } } ignore_live_leader: false dest_uuid: "90487c84b53440daa01a5039b2653a72"
I20250903 08:11:11.359256 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 23 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:11.359668 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8; no voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:11.359800 13350 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20250903 08:11:11.362113 13249 raft_consensus.cc:1238] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 18 FOLLOWER]: Rejecting Update request from peer e6ce9b384db14226b80d2025e5cc34df for earlier term 17. Current term is 18. Ops: []
I20250903 08:11:11.362464 13341 consensus_queue.cc:1046] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 }, Status: INVALID_TERM, Last received: 17.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20250903 08:11:11.362622 13346 raft_consensus.cc:3053] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 17 LEADER]: Stepping down as leader of term 17
I20250903 08:11:11.362680 13346 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6ce9b384db14226b80d2025e5cc34df, State: Running, Role: LEADER
I20250903 08:11:11.362795 13346 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 17.23, Last appended by leader: 23, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.362934 13346 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 17 FOLLOWER]: Advancing to term 18
I20250903 08:11:11.406987 12408 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 1 LEADER]: Raft consensus shutting down.
I20250903 08:11:11.407151 12408 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:11.407202 12408 tablet_replica.cc:331] T 00000000000000000000000000000000 P 351552a25c8a43d0a65ade4963196ddb: stopping tablet replica
I20250903 08:11:11.419893 12408 master.cc:583] Master@127.12.30.62:34051 shutdown complete.
I20250903 08:11:11.422365 12408 tablet_server.cc:178] TabletServer@127.12.30.1:0 shutting down...
I20250903 08:11:11.456375 13343 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6ce9b384db14226b80d2025e5cc34df)
I20250903 08:11:11.456480 13343 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 17 FOLLOWER]: Advancing to term 18
I20250903 08:11:11.457340 13343 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.457713 13343 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 18 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
I20250903 08:11:11.458106 13174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 18 candidate_status { last_received { term: 17 index: 23 } } ignore_live_leader: false dest_uuid: "e6ce9b384db14226b80d2025e5cc34df"
I20250903 08:11:11.458106 13249 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c7d95e51d9014fb1b067d05e736f07a1" candidate_uuid: "90487c84b53440daa01a5039b2653a72" candidate_term: 18 candidate_status { last_received { term: 17 index: 23 } } ignore_live_leader: false dest_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8"
I20250903 08:11:11.458273 13249 raft_consensus.cc:2391] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 90487c84b53440daa01a5039b2653a72 in current term 18: Already voted for candidate 3599a09f95cd4e08aafaa4bbe02be4d8 in this term.
I20250903 08:11:11.458971 13174 raft_consensus.cc:2466] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90487c84b53440daa01a5039b2653a72 in term 18.
I20250903 08:11:11.459223 13061 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8
I20250903 08:11:11.459363 13343 raft_consensus.cc:2802] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Leader election won for term 18
I20250903 08:11:11.459451 13343 raft_consensus.cc:695] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 LEADER]: Becoming Leader. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Running, Role: LEADER
I20250903 08:11:11.459585 13343 consensus_queue.cc:237] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 17.23, Last appended by leader: 23, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.465435 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:11.465566 12408 tablet_replica.cc:331] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: stopping tablet replica
I20250903 08:11:11.465679 12408 raft_consensus.cc:2241] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 LEADER]: Raft consensus shutting down.
I20250903 08:11:11.465778 12408 pending_rounds.cc:70] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Trying to abort 1 pending ops.
I20250903 08:11:11.465818 12408 pending_rounds.cc:77] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Aborting op as it isn't in flight: id { term: 18 index: 24 } timestamp: 7196209444699799552 op_type: NO_OP noop_request { }
I20250903 08:11:11.465866 12408 raft_consensus.cc:2887] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250903 08:11:11.465907 12408 raft_consensus.cc:2270] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:11.468292 12408 tablet_server.cc:195] TabletServer@127.12.30.1:0 shutdown complete.
I20250903 08:11:11.469981 12408 tablet_server.cc:178] TabletServer@127.12.30.2:0 shutting down...
I20250903 08:11:11.485762 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:11.485908 12408 tablet_replica.cc:331] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df: stopping tablet replica
I20250903 08:11:11.486001 12408 raft_consensus.cc:2241] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 18 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:11.486084 12408 raft_consensus.cc:2270] T c7d95e51d9014fb1b067d05e736f07a1 P e6ce9b384db14226b80d2025e5cc34df [term 18 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:11.488284 12408 tablet_server.cc:195] TabletServer@127.12.30.2:0 shutdown complete.
I20250903 08:11:11.489799 12408 tablet_server.cc:178] TabletServer@127.12.30.3:0 shutting down...
I20250903 08:11:11.499398 13350 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:11.499464 13350 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 18 FOLLOWER]: Advancing to term 19
I20250903 08:11:11.500157 13350 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.500456 13350 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 19 election: Requested vote from peers 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
W20250903 08:11:11.501621 13212 leader_election.cc:336] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 90487c84b53440daa01a5039b2653a72 (127.12.30.1:45485): Network error: Client connection negotiation failed: client connection to 127.12.30.1:45485: connect: Connection refused (error 111)
W20250903 08:11:11.501726 13212 leader_election.cc:336] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243): Network error: Client connection negotiation failed: client connection to 127.12.30.2:45243: connect: Connection refused (error 111)
I20250903 08:11:11.501776 13212 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3599a09f95cd4e08aafaa4bbe02be4d8; no voters: 90487c84b53440daa01a5039b2653a72, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:11.501890 13350 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250903 08:11:11.506305 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:11.506467 12408 tablet_replica.cc:331] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8: stopping tablet replica
I20250903 08:11:11.506593 12408 raft_consensus.cc:2241] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 19 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:11.506688 12408 raft_consensus.cc:2270] T c7d95e51d9014fb1b067d05e736f07a1 P 3599a09f95cd4e08aafaa4bbe02be4d8 [term 19 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:11.518784 12408 tablet_server.cc:195] TabletServer@127.12.30.3:0 shutdown complete.
I20250903 08:11:11.520234 12408 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20250903 08:11:11.520453 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:11.522171 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.522318 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.522454 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:11.522255 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.522630 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:11.522668 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:11.522682 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887071522682 us; error 0 us; skew 500 ppm
I20250903 08:11:11.523361 12408 webserver.cc:480] Webserver started at http://127.12.30.1:35907/ using document root <none> and password file <none>
I20250903 08:11:11.523475 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:11.523505 12408 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250903 08:11:11.524452 12408 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:11.526741 13366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.527040 12408 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250903 08:11:11.527100 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root
uuid: "90487c84b53440daa01a5039b2653a72"
format_stamp: "Formatted at 2025-09-03 08:11:07 on dist-test-slave-d521"
I20250903 08:11:11.527155 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1756887064782267-12408-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5273
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:11.537420 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:11.537649 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:11.537853 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:11.538306 13373 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250903 08:11:11.539251 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250903 08:11:11.539292 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:11.539322 12408 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250903 08:11:11.539942 12408 ts_tablet_manager.cc:610] Registered 1 tablets
I20250903 08:11:11.539979 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:11.540041 13373 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Bootstrap starting.
I20250903 08:11:11.544348 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.1:45485
I20250903 08:11:11.544530 13373 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=1 applied=8 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250903 08:11:11.545063 13373 tablet_bootstrap.cc:492] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Bootstrap complete.
I20250903 08:11:11.545199 13431 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.1:45485 every 8 connection(s)
I20250903 08:11:11.545251 13373 ts_tablet_manager.cc:1397] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250903 08:11:11.546191 13373 raft_consensus.cc:357] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.546432 13373 raft_consensus.cc:738] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90487c84b53440daa01a5039b2653a72, State: Initialized, Role: FOLLOWER
I20250903 08:11:11.546674 13373 consensus_queue.cc:260] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 23, Last appended: 18.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.547297 13373 ts_tablet_manager.cc:1428] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250903 08:11:11.548621 13432 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20250903 08:11:11.560678 12408 tablet_server.cc:178] TabletServer@127.12.30.1:45485 shutting down...
I20250903 08:11:11.646344 13437 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:11.646479 13437 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 18 FOLLOWER]: Advancing to term 19
I20250903 08:11:11.647464 13437 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.647993 13437 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 19 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
W20250903 08:11:11.649052 13369 leader_election.cc:336] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321): Network error: Client connection negotiation failed: client connection to 127.12.30.3:41321: connect: Connection refused (error 111)
W20250903 08:11:11.649134 13367 leader_election.cc:336] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243): Network error: Client connection negotiation failed: client connection to 127.12.30.2:45243: connect: Connection refused (error 111)
I20250903 08:11:11.649217 13367 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90487c84b53440daa01a5039b2653a72; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:11.649338 13437 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250903 08:11:11.775961 13437 raft_consensus.cc:491] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:11.776087 13437 raft_consensus.cc:3058] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 19 FOLLOWER]: Advancing to term 20
I20250903 08:11:11.776999 13437 raft_consensus.cc:513] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3599a09f95cd4e08aafaa4bbe02be4d8" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 41321 } } peers { permanent_uuid: "90487c84b53440daa01a5039b2653a72" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 45485 } } peers { permanent_uuid: "e6ce9b384db14226b80d2025e5cc34df" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 45243 } }
I20250903 08:11:11.777401 13437 leader_election.cc:290] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 20 election: Requested vote from peers 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321), e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243)
W20250903 08:11:11.778402 13369 leader_election.cc:336] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 3599a09f95cd4e08aafaa4bbe02be4d8 (127.12.30.3:41321): Network error: Client connection negotiation failed: client connection to 127.12.30.3:41321: connect: Connection refused (error 111)
W20250903 08:11:11.778486 13367 leader_election.cc:336] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer e6ce9b384db14226b80d2025e5cc34df (127.12.30.2:45243): Network error: Client connection negotiation failed: client connection to 127.12.30.2:45243: connect: Connection refused (error 111)
I20250903 08:11:11.778575 13367 leader_election.cc:304] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90487c84b53440daa01a5039b2653a72; no voters: 3599a09f95cd4e08aafaa4bbe02be4d8, e6ce9b384db14226b80d2025e5cc34df
I20250903 08:11:11.778714 13437 raft_consensus.cc:2747] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20250903 08:11:11.795845 12408 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250903 08:11:11.796110 12408 tablet_replica.cc:331] stopping tablet replica
I20250903 08:11:11.796216 12408 raft_consensus.cc:2241] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 20 FOLLOWER]: Raft consensus shutting down.
I20250903 08:11:11.796298 12408 pending_rounds.cc:70] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Trying to abort 1 pending ops.
I20250903 08:11:11.796321 12408 pending_rounds.cc:77] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72: Aborting op as it isn't in flight: id { term: 18 index: 24 } timestamp: 7196209444699799552 op_type: NO_OP noop_request { }
I20250903 08:11:11.796376 12408 raft_consensus.cc:2887] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250903 08:11:11.796418 12408 raft_consensus.cc:2270] T c7d95e51d9014fb1b067d05e736f07a1 P 90487c84b53440daa01a5039b2653a72 [term 20 FOLLOWER]: Raft consensus is shut down!
I20250903 08:11:11.808972 12408 tablet_server.cc:195] TabletServer@127.12.30.1:45485 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (3931 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20250903 08:11:11.814970 12408 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.12.30.62:34267
I20250903 08:11:11.815277 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:11.816689 13441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.816797 13443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.816876 13440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.817034 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:11.817173 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:11.817212 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:11.817234 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887071817234 us; error 0 us; skew 500 ppm
I20250903 08:11:11.817986 12408 webserver.cc:480] Webserver started at http://127.12.30.62:44969/ using document root <none> and password file <none>
I20250903 08:11:11.818125 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:11.818186 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:11.818259 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:11.818625 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/master-0-root/instance:
uuid: "7be24fb8961c485893d0a8ecc9353121"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.819957 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:11.820772 13448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.821025 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20250903 08:11:11.821092 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/master-0-root
uuid: "7be24fb8961c485893d0a8ecc9353121"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.821162 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:11.826632 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:11.826916 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:11.830953 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.62:34267
I20250903 08:11:11.831319 13500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.62:34267 every 8 connection(s)
I20250903 08:11:11.831801 13501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:11.833365 13501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121: Bootstrap starting.
I20250903 08:11:11.834038 13501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:11.834792 13501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121: No bootstrap required, opened a new log
I20250903 08:11:11.835104 13501 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be24fb8961c485893d0a8ecc9353121" member_type: VOTER }
I20250903 08:11:11.835180 13501 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:11.835228 13501 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be24fb8961c485893d0a8ecc9353121, State: Initialized, Role: FOLLOWER
I20250903 08:11:11.835366 13501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be24fb8961c485893d0a8ecc9353121" member_type: VOTER }
I20250903 08:11:11.835428 13501 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250903 08:11:11.835456 13501 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250903 08:11:11.835489 13501 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:11.836047 13501 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be24fb8961c485893d0a8ecc9353121" member_type: VOTER }
I20250903 08:11:11.836143 13501 leader_election.cc:304] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7be24fb8961c485893d0a8ecc9353121; no voters:
I20250903 08:11:11.836277 13501 leader_election.cc:290] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250903 08:11:11.836359 13504 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:11:11.836547 13504 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [term 1 LEADER]: Becoming Leader. State: Replica: 7be24fb8961c485893d0a8ecc9353121, State: Running, Role: LEADER
I20250903 08:11:11.836651 13501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [sys.catalog]: configured and running, proceeding with master startup.
I20250903 08:11:11.836694 13504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be24fb8961c485893d0a8ecc9353121" member_type: VOTER }
I20250903 08:11:11.837033 13505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7be24fb8961c485893d0a8ecc9353121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be24fb8961c485893d0a8ecc9353121" member_type: VOTER } }
I20250903 08:11:11.837131 13505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [sys.catalog]: This master's current role is: LEADER
I20250903 08:11:11.837042 13506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be24fb8961c485893d0a8ecc9353121. Latest consensus state: current_term: 1 leader_uuid: "7be24fb8961c485893d0a8ecc9353121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be24fb8961c485893d0a8ecc9353121" member_type: VOTER } }
I20250903 08:11:11.837397 13506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121 [sys.catalog]: This master's current role is: LEADER
I20250903 08:11:11.837651 13509 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250903 08:11:11.838364 13509 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250903 08:11:11.838548 12408 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250903 08:11:11.839818 13509 catalog_manager.cc:1349] Generated new cluster ID: 8360888c7f8e43bb926c88031765eb55
I20250903 08:11:11.839895 13509 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250903 08:11:11.847816 13509 catalog_manager.cc:1372] Generated new certificate authority record
I20250903 08:11:11.848296 13509 catalog_manager.cc:1506] Loading token signing keys...
I20250903 08:11:11.852393 13509 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7be24fb8961c485893d0a8ecc9353121: Generated new TSK 0
I20250903 08:11:11.852519 13509 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250903 08:11:11.854391 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:11.856027 13525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.856088 12408 server_base.cc:1047] running on GCE node
W20250903 08:11:11.856189 13523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.856031 13522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.856422 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:11.856464 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:11.856487 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887071856487 us; error 0 us; skew 500 ppm
I20250903 08:11:11.857267 12408 webserver.cc:480] Webserver started at http://127.12.30.1:46699/ using document root <none> and password file <none>
I20250903 08:11:11.857426 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:11.857474 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:11.857554 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:11.857885 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-0-root/instance:
uuid: "76db1b58a0ff42828da1332775559677"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.859202 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20250903 08:11:11.860145 13530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.860440 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:11.860500 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-0-root
uuid: "76db1b58a0ff42828da1332775559677"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.860621 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:11.873955 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:11.874182 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:11.874366 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:11.874738 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:11.874771 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.874801 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:11.874833 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.878556 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.1:42065
I20250903 08:11:11.878577 13593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.1:42065 every 8 connection(s)
I20250903 08:11:11.879268 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:11.880856 13598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.880867 13601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.880959 13599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.881315 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:11.881446 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:11.881490 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:11.881511 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887071881511 us; error 0 us; skew 500 ppm
I20250903 08:11:11.882316 12408 webserver.cc:480] Webserver started at http://127.12.30.2:36163/ using document root <none> and password file <none>
I20250903 08:11:11.882470 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:11.882532 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:11.882614 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:11.882989 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-1-root/instance:
uuid: "959f3118f3104904afb42b2c7e870120"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.883476 13594 heartbeater.cc:344] Connected to a master server at 127.12.30.62:34267
I20250903 08:11:11.883553 13594 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:11.883734 13594 heartbeater.cc:507] Master 127.12.30.62:34267 requested a full tablet report, sending...
I20250903 08:11:11.884281 13465 ts_manager.cc:194] Registered new tserver with Master: 76db1b58a0ff42828da1332775559677 (127.12.30.1:42065)
I20250903 08:11:11.884502 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20250903 08:11:11.885175 13465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43014
I20250903 08:11:11.885423 13606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.885668 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:11.885730 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-1-root
uuid: "959f3118f3104904afb42b2c7e870120"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.885789 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:11.902143 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:11.902410 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:11.902594 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:11.902941 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:11.902974 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.903019 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:11.903048 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.906585 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.2:33745
I20250903 08:11:11.906612 13669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.2:33745 every 8 connection(s)
I20250903 08:11:11.907243 12408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250903 08:11:11.909003 13676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.908979 13674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250903 08:11:11.908993 13673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250903 08:11:11.909214 12408 server_base.cc:1047] running on GCE node
I20250903 08:11:11.909358 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250903 08:11:11.909392 12408 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250903 08:11:11.909415 12408 hybrid_clock.cc:648] HybridClock initialized: now 1756887071909415 us; error 0 us; skew 500 ppm
I20250903 08:11:11.910207 12408 webserver.cc:480] Webserver started at http://127.12.30.3:39127/ using document root <none> and password file <none>
I20250903 08:11:11.910360 12408 fs_manager.cc:362] Metadata directory not provided
I20250903 08:11:11.910418 12408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250903 08:11:11.910493 12408 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250903 08:11:11.910948 12408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-2-root/instance:
uuid: "94ad0afff8884089964ddb23398f9b9c"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.911005 13670 heartbeater.cc:344] Connected to a master server at 127.12.30.62:34267
I20250903 08:11:11.911089 13670 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:11.911255 13670 heartbeater.cc:507] Master 127.12.30.62:34267 requested a full tablet report, sending...
I20250903 08:11:11.911737 13465 ts_manager.cc:194] Registered new tserver with Master: 959f3118f3104904afb42b2c7e870120 (127.12.30.2:33745)
I20250903 08:11:11.912178 13465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43020
I20250903 08:11:11.912547 12408 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:11.913415 13681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.913618 12408 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250903 08:11:11.913681 12408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-2-root
uuid: "94ad0afff8884089964ddb23398f9b9c"
format_stamp: "Formatted at 2025-09-03 08:11:11 on dist-test-slave-d521"
I20250903 08:11:11.913743 12408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskilJnYZ/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1756887064782267-12408-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250903 08:11:11.920933 12408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250903 08:11:11.921240 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250903 08:11:11.921463 12408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250903 08:11:11.921799 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250903 08:11:11.921833 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.921875 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250903 08:11:11.921905 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250903 08:11:11.926098 12408 rpc_server.cc:307] RPC server started. Bound to: 127.12.30.3:34709
I20250903 08:11:11.926162 13744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.12.30.3:34709 every 8 connection(s)
I20250903 08:11:11.930359 13745 heartbeater.cc:344] Connected to a master server at 127.12.30.62:34267
I20250903 08:11:11.930436 13745 heartbeater.cc:461] Registering TS with master...
I20250903 08:11:11.930583 13745 heartbeater.cc:507] Master 127.12.30.62:34267 requested a full tablet report, sending...
I20250903 08:11:11.931056 13465 ts_manager.cc:194] Registered new tserver with Master: 94ad0afff8884089964ddb23398f9b9c (127.12.30.3:34709)
I20250903 08:11:11.931473 13465 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43032
I20250903 08:11:11.932035 12408 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.0055475s
I20250903 08:11:11.940552 12408 test_util.cc:276] Using random seed: 634740680
I20250903 08:11:11.946314 13465 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43050:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250903 08:11:11.946735 13465 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250903 08:11:11.955001 13558 tablet_service.cc:1468] Processing CreateTablet for tablet f8bee6cbe1db4d04841d99d1a8edb8ca (DEFAULT_TABLE table=test-workload [id=ff40e30c40e04bceb25221646c8df796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:11.955025 13709 tablet_service.cc:1468] Processing CreateTablet for tablet f8bee6cbe1db4d04841d99d1a8edb8ca (DEFAULT_TABLE table=test-workload [id=ff40e30c40e04bceb25221646c8df796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:11.955235 13558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8bee6cbe1db4d04841d99d1a8edb8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:11.955260 13709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8bee6cbe1db4d04841d99d1a8edb8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:11.955272 13634 tablet_service.cc:1468] Processing CreateTablet for tablet f8bee6cbe1db4d04841d99d1a8edb8ca (DEFAULT_TABLE table=test-workload [id=ff40e30c40e04bceb25221646c8df796]), partition=RANGE (key) PARTITION UNBOUNDED
I20250903 08:11:11.955483 13634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8bee6cbe1db4d04841d99d1a8edb8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250903 08:11:11.957187 13766 tablet_bootstrap.cc:492] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677: Bootstrap starting.
I20250903 08:11:11.957533 13768 tablet_bootstrap.cc:492] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120: Bootstrap starting.
I20250903 08:11:11.957648 13767 tablet_bootstrap.cc:492] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c: Bootstrap starting.
I20250903 08:11:11.958214 13766 tablet_bootstrap.cc:654] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:11.958654 13767 tablet_bootstrap.cc:654] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:11.958654 13768 tablet_bootstrap.cc:654] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120: Neither blocks nor log segments found. Creating new log.
I20250903 08:11:11.959189 13766 tablet_bootstrap.cc:492] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677: No bootstrap required, opened a new log
I20250903 08:11:11.959316 13766 ts_tablet_manager.cc:1397] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:11:11.959785 13767 tablet_bootstrap.cc:492] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c: No bootstrap required, opened a new log
I20250903 08:11:11.959761 13766 raft_consensus.cc:357] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:11.959874 13768 tablet_bootstrap.cc:492] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120: No bootstrap required, opened a new log
I20250903 08:11:11.959928 13767 ts_tablet_manager.cc:1397] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:11:11.959978 13768 ts_tablet_manager.cc:1397] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250903 08:11:11.960281 13767 raft_consensus.cc:357] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:11.960362 13767 raft_consensus.cc:383] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:11.960336 13768 raft_consensus.cc:357] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:11.960410 13766 raft_consensus.cc:383] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:11.960431 13768 raft_consensus.cc:383] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250903 08:11:11.960446 13766 raft_consensus.cc:738] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76db1b58a0ff42828da1332775559677, State: Initialized, Role: FOLLOWER
I20250903 08:11:11.960479 13768 raft_consensus.cc:738] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 959f3118f3104904afb42b2c7e870120, State: Initialized, Role: FOLLOWER
I20250903 08:11:11.960592 13766 consensus_queue.cc:260] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:11.960623 13768 consensus_queue.cc:260] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:11.961032 13767 raft_consensus.cc:738] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ad0afff8884089964ddb23398f9b9c, State: Initialized, Role: FOLLOWER
I20250903 08:11:11.961014 13670 heartbeater.cc:499] Master 127.12.30.62:34267 was elected leader, sending a full tablet report...
I20250903 08:11:11.961005 13594 heartbeater.cc:499] Master 127.12.30.62:34267 was elected leader, sending a full tablet report...
I20250903 08:11:11.961179 13767 consensus_queue.cc:260] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:11.961520 13745 heartbeater.cc:499] Master 127.12.30.62:34267 was elected leader, sending a full tablet report...
I20250903 08:11:11.961011 13766 ts_tablet_manager.cc:1428] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:11:11.961020 13768 ts_tablet_manager.cc:1428] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250903 08:11:11.962146 13767 ts_tablet_manager.cc:1428] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250903 08:11:12.110572 13774 raft_consensus.cc:491] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250903 08:11:12.110718 13774 raft_consensus.cc:513] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:12.111227 13774 leader_election.cc:290] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76db1b58a0ff42828da1332775559677 (127.12.30.1:42065), 959f3118f3104904afb42b2c7e870120 (127.12.30.2:33745)
I20250903 08:11:12.114655 13644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8bee6cbe1db4d04841d99d1a8edb8ca" candidate_uuid: "94ad0afff8884089964ddb23398f9b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959f3118f3104904afb42b2c7e870120" is_pre_election: true
I20250903 08:11:12.114658 13568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8bee6cbe1db4d04841d99d1a8edb8ca" candidate_uuid: "94ad0afff8884089964ddb23398f9b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76db1b58a0ff42828da1332775559677" is_pre_election: true
I20250903 08:11:12.114804 13644 raft_consensus.cc:2466] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ad0afff8884089964ddb23398f9b9c in term 0.
I20250903 08:11:12.114858 13568 raft_consensus.cc:2466] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94ad0afff8884089964ddb23398f9b9c in term 0.
I20250903 08:11:12.115099 13685 leader_election.cc:304] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94ad0afff8884089964ddb23398f9b9c, 959f3118f3104904afb42b2c7e870120; no voters:
I20250903 08:11:12.115271 13774 raft_consensus.cc:2802] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250903 08:11:12.115326 13774 raft_consensus.cc:491] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250903 08:11:12.115355 13774 raft_consensus.cc:3058] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:12.116173 13774 raft_consensus.cc:513] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:12.116523 13774 leader_election.cc:290] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [CANDIDATE]: Term 1 election: Requested vote from peers 76db1b58a0ff42828da1332775559677 (127.12.30.1:42065), 959f3118f3104904afb42b2c7e870120 (127.12.30.2:33745)
I20250903 08:11:12.116822 13644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8bee6cbe1db4d04841d99d1a8edb8ca" candidate_uuid: "94ad0afff8884089964ddb23398f9b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "959f3118f3104904afb42b2c7e870120"
I20250903 08:11:12.116828 13568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8bee6cbe1db4d04841d99d1a8edb8ca" candidate_uuid: "94ad0afff8884089964ddb23398f9b9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76db1b58a0ff42828da1332775559677"
I20250903 08:11:12.116927 13644 raft_consensus.cc:3058] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:12.116930 13568 raft_consensus.cc:3058] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 0 FOLLOWER]: Advancing to term 1
I20250903 08:11:12.117749 13644 raft_consensus.cc:2466] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ad0afff8884089964ddb23398f9b9c in term 1.
I20250903 08:11:12.117755 13568 raft_consensus.cc:2466] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94ad0afff8884089964ddb23398f9b9c in term 1.
I20250903 08:11:12.118012 13684 leader_election.cc:304] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76db1b58a0ff42828da1332775559677, 94ad0afff8884089964ddb23398f9b9c; no voters:
I20250903 08:11:12.118222 13774 raft_consensus.cc:2802] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 1 FOLLOWER]: Leader election won for term 1
I20250903 08:11:12.118417 13774 raft_consensus.cc:695] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [term 1 LEADER]: Becoming Leader. State: Replica: 94ad0afff8884089964ddb23398f9b9c, State: Running, Role: LEADER
I20250903 08:11:12.118566 13774 consensus_queue.cc:237] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } }
I20250903 08:11:12.120023 13465 catalog_manager.cc:5582] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 94ad0afff8884089964ddb23398f9b9c (127.12.30.3). New cstate: current_term: 1 leader_uuid: "94ad0afff8884089964ddb23398f9b9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94ad0afff8884089964ddb23398f9b9c" member_type: VOTER last_known_addr { host: "127.12.30.3" port: 34709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 } health_report { overall_health: UNKNOWN } } }
I20250903 08:11:12.145640 13644 raft_consensus.cc:1273] T f8bee6cbe1db4d04841d99d1a8edb8ca P 959f3118f3104904afb42b2c7e870120 [term 1 FOLLOWER]: Refusing update from remote peer 94ad0afff8884089964ddb23398f9b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:11:12.145764 13568 raft_consensus.cc:1273] T f8bee6cbe1db4d04841d99d1a8edb8ca P 76db1b58a0ff42828da1332775559677 [term 1 FOLLOWER]: Refusing update from remote peer 94ad0afff8884089964ddb23398f9b9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250903 08:11:12.146114 13776 consensus_queue.cc:1035] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "959f3118f3104904afb42b2c7e870120" member_type: VOTER last_known_addr { host: "127.12.30.2" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250903 08:11:12.146328 13774 consensus_queue.cc:1035] T f8bee6cbe1db4d04841d99d1a8edb8ca P 94ad0afff8884089964ddb23398f9b9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "76db1b58a0ff42828da1332775559677" member_type: VOTER last_known_addr { host: "127.12.30.1" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
F20250903 08:11:12.178184 13788 log_util.cc:806] Check failed: !IsFooterWritten()
*** Check failure stack trace: ***
*** Aborted at 1756887072 (unix time) try "date -d @1756887072" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003078) received by PID 12408 (TID 0x7f4ee6dd3700) from PID 12408; stack trace: ***
@ 0x7f4f9914b980 (unknown) at ??:0
@ 0x7f4f969defb7 gsignal at ??:0
@ 0x7f4f969e0921 abort at ??:0
@ 0x7f4f97be5dcd google::LogMessage::Fail() at ??:0
@ 0x7f4f97be9b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4f97be57cc google::LogMessage::Flush() at ??:0
@ 0x7f4f97be6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4f997ce611 kudu::log::WritableLogSegment::WriteFooter() at ??:0
@ 0x7f4f997dfc0a kudu::log::SegmentAllocator::FinishCurrentSegment() at ??:0
@ 0x7f4f997e21b4 kudu::log::SegmentAllocator::RollOver() at ??:0
@ 0x7f4f997ded90 kudu::log::SegmentAllocator::AllocateOrRollOverIfNecessary() at ??:0
@ 0x7f4f997e431d kudu::log::Log::WriteBatch() at ??:0
@ 0x7f4f997dd849 kudu::log::Log::AppendThread::HandleBatches() at ??:0
@ 0x7f4f997dd284 kudu::log::Log::AppendThread::ProcessQueue() at ??:0
@ 0x7f4f997dc91b _ZZN4kudu3log3Log12AppendThread4WakeEvENKUlvE_clEv at ??:0
@ 0x7f4f997e9642 _ZNSt17_Function_handlerIFvvEZN4kudu3log3Log12AppendThread4WakeEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556f59468346 std::function<>::operator()() at ??:0
@ 0x7f4f989609c1 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4f989612ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4f98962a3b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x556f59468346 std::function<>::operator()() at ??:0
@ 0x7f4f98950924 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f991406db start_thread at ??:0
@ 0x7f4f96ac171f clone at ??:0