[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TimestampAdvancementITest
[ RUN ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20261117 09:06:31.337812 21969 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.116.126:35417
I20261117 09:06:31.338104 21969 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261117 09:06:31.338238 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:31.341805 21975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.341813 21977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.341813 21974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:31.341838 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:31.342144 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:31.342180 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:31.342216 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906391342215 us; error 0 us; skew 500 ppm
I20261117 09:06:31.343101 21969 webserver.cc:577] Webserver started at http://127.21.116.126:45257/ using document root <none> and password file <none>
I20261117 09:06:31.343225 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:31.343271 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:31.343340 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:31.344292 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/master-0-root/instance:
uuid: "1c1609560ec2400ca45db3811b388372"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.345345 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20261117 09:06:31.345856 21982 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.345999 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.346051 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/master-0-root
uuid: "1c1609560ec2400ca45db3811b388372"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.346114 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:31.362315 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:31.362531 21969 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261117 09:06:31.362617 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:31.362905 21988 maintenance_manager.cc:346] Maintenance manager is disabled (check --enable_maintenance_manager).
I20261117 09:06:31.366199 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.126:35417
I20261117 09:06:31.366248 22034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.126:35417 every 8 connection(s)
I20261117 09:06:31.366662 22035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:31.368722 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372: Bootstrap starting.
I20261117 09:06:31.369426 22035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:31.369635 22035 log.cc:826] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372: Log is configured to *not* fsync() on all Append() calls
I20261117 09:06:31.370075 22035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372: No bootstrap required, opened a new log
I20261117 09:06:31.371222 22035 raft_consensus.cc:360] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1609560ec2400ca45db3811b388372" member_type: VOTER }
I20261117 09:06:31.371300 22035 raft_consensus.cc:386] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:31.371317 22035 raft_consensus.cc:748] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1609560ec2400ca45db3811b388372, State: Initialized, Role: FOLLOWER
I20261117 09:06:31.371409 22035 consensus_queue.cc:261] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1609560ec2400ca45db3811b388372" member_type: VOTER }
I20261117 09:06:31.371454 22035 raft_consensus.cc:400] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:31.371481 22035 raft_consensus.cc:499] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:31.371518 22035 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:31.371953 22035 raft_consensus.cc:521] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1609560ec2400ca45db3811b388372" member_type: VOTER }
I20261117 09:06:31.372028 22035 leader_election.cc:304] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c1609560ec2400ca45db3811b388372; no voters:
I20261117 09:06:31.372150 22035 leader_election.cc:290] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261117 09:06:31.372189 22038 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:31.372318 22038 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 1 LEADER]: Becoming Leader. State: Replica: 1c1609560ec2400ca45db3811b388372, State: Running, Role: LEADER
I20261117 09:06:31.372328 22035 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [sys.catalog]: configured and running, proceeding with master startup.
I20261117 09:06:31.372406 22038 consensus_queue.cc:238] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1609560ec2400ca45db3811b388372" member_type: VOTER }
I20261117 09:06:31.372682 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c1609560ec2400ca45db3811b388372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1609560ec2400ca45db3811b388372" member_type: VOTER } }
I20261117 09:06:31.372706 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c1609560ec2400ca45db3811b388372. Latest consensus state: current_term: 1 leader_uuid: "1c1609560ec2400ca45db3811b388372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c1609560ec2400ca45db3811b388372" member_type: VOTER } }
I20261117 09:06:31.372814 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:31.372802 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:31.373034 22045 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261117 09:06:31.373401 22045 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261117 09:06:31.373597 21969 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261117 09:06:31.375361 22045 catalog_manager.cc:1398] Generated new cluster ID: cf4768142dc6484398b764b098d8b69b
I20261117 09:06:31.375419 22045 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261117 09:06:31.394973 22045 catalog_manager.cc:1421] Generated new certificate authority record
I20261117 09:06:31.395516 22045 catalog_manager.cc:1555] Loading token signing keys...
I20261117 09:06:31.400074 22045 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372: Generated new TSK 0
I20261117 09:06:31.400203 22045 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261117 09:06:31.405295 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:31.405431 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:31.406533 22058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.406566 22057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.406760 22060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:31.410763 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:31.410856 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:31.410882 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:31.410916 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906391410916 us; error 0 us; skew 500 ppm
I20261117 09:06:31.411432 21969 webserver.cc:577] Webserver started at http://127.21.116.65:44771/ using document root <none> and password file <none>
I20261117 09:06:31.411512 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:31.411553 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:31.411595 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:31.411810 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root/instance:
uuid: "224ffdac0c8c486d912551a7a37ddfe4"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.412596 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:31.413053 22065 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.413190 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.413231 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "224ffdac0c8c486d912551a7a37ddfe4"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.413271 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:31.424000 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:31.424213 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:31.424355 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:31.424587 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:31.424614 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.424638 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:31.424655 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.428450 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:33161
I20261117 09:06:31.428514 22128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:33161 every 8 connection(s)
I20261117 09:06:31.428762 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:31.428853 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:31.430007 22131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.430042 22132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.430511 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:31.430896 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:31.430998 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:31.431027 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:31.431043 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906391431043 us; error 0 us; skew 500 ppm
I20261117 09:06:31.431542 21969 webserver.cc:577] Webserver started at http://127.21.116.66:36797/ using document root <none> and password file <none>
I20261117 09:06:31.431622 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:31.431663 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:31.431701 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:31.431921 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-1-root/instance:
uuid: "44d43c0419874c329f139eea1012c23e"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.432744 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:31.433190 22141 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.433292 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.433326 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-1-root
uuid: "44d43c0419874c329f139eea1012c23e"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.433367 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:31.434916 22129 heartbeater.cc:345] Connected to a master server at 127.21.116.126:35417
I20261117 09:06:31.435001 22129 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:31.435151 22129 heartbeater.cc:508] Master 127.21.116.126:35417 requested a full tablet report, sending...
I20261117 09:06:31.435568 21999 ts_manager.cc:210] Registered new tserver with Master: 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:31.436497 21999 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37100
I20261117 09:06:31.444864 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:31.445062 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:31.445194 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:31.445364 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:31.445389 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.445413 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:31.445436 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.448740 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.66:41917
I20261117 09:06:31.448782 22204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.66:41917 every 8 connection(s)
I20261117 09:06:31.449133 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:31.449232 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:31.450235 22209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:31.450639 22208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:31.451025 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:31.451843 22211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:31.452001 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:31.452031 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:31.452049 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906391452049 us; error 0 us; skew 500 ppm
I20261117 09:06:31.452593 21969 webserver.cc:577] Webserver started at http://127.21.116.67:44579/ using document root <none> and password file <none>
I20261117 09:06:31.452669 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:31.452709 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:31.452770 22205 heartbeater.cc:345] Connected to a master server at 127.21.116.126:35417
I20261117 09:06:31.452775 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:31.452848 22205 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:31.452957 22205 heartbeater.cc:508] Master 127.21.116.126:35417 requested a full tablet report, sending...
I20261117 09:06:31.453044 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-2-root/instance:
uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.453181 21999 ts_manager.cc:210] Registered new tserver with Master: 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:31.453634 21999 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37110
I20261117 09:06:31.453868 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:31.454329 22216 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.454458 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.454504 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-2-root
uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:31.454545 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:31.460960 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:31.461148 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:31.461274 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:31.461443 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:31.461468 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.461491 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:31.461514 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:31.464781 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.67:40241
I20261117 09:06:31.464826 22279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.67:40241 every 8 connection(s)
I20261117 09:06:31.467875 22280 heartbeater.cc:345] Connected to a master server at 127.21.116.126:35417
I20261117 09:06:31.467923 22280 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:31.468006 22280 heartbeater.cc:508] Master 127.21.116.126:35417 requested a full tablet report, sending...
I20261117 09:06:31.468245 21999 ts_manager.cc:210] Registered new tserver with Master: abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241)
I20261117 09:06:31.468297 21969 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003347421s
I20261117 09:06:31.468726 21999 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37124
I20261117 09:06:31.482743 21999 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37142:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261117 09:06:31.482961 21999 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261117 09:06:31.489012 22093 tablet_service.cc:1500] Processing CreateTablet for tablet c62d3ed753144c988a887b52c9eb4491 (DEFAULT_TABLE table=test-workload [id=558e5657c2c44fb2a54c6e32184df7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:31.489019 22244 tablet_service.cc:1500] Processing CreateTablet for tablet c62d3ed753144c988a887b52c9eb4491 (DEFAULT_TABLE table=test-workload [id=558e5657c2c44fb2a54c6e32184df7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:31.489172 22093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62d3ed753144c988a887b52c9eb4491. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:31.489173 22244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62d3ed753144c988a887b52c9eb4491. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:31.490553 22169 tablet_service.cc:1500] Processing CreateTablet for tablet c62d3ed753144c988a887b52c9eb4491 (DEFAULT_TABLE table=test-workload [id=558e5657c2c44fb2a54c6e32184df7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:31.490646 22169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c62d3ed753144c988a887b52c9eb4491. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:31.491205 22302 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901: Bootstrap starting.
I20261117 09:06:31.491755 22303 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e: Bootstrap starting.
I20261117 09:06:31.492237 22303 tablet_bootstrap.cc:654] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:31.491767 22302 tablet_bootstrap.cc:654] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:31.492841 22303 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e: No bootstrap required, opened a new log
I20261117 09:06:31.492933 22303 ts_tablet_manager.cc:1449] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:31.493093 22302 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901: No bootstrap required, opened a new log
I20261117 09:06:31.493103 22303 raft_consensus.cc:360] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.493156 22302 ts_tablet_manager.cc:1449] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261117 09:06:31.493407 22302 raft_consensus.cc:360] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.493221 22303 raft_consensus.cc:386] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:31.493551 22302 raft_consensus.cc:386] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:31.493568 22303 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Initialized, Role: FOLLOWER
I20261117 09:06:31.493580 22302 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe59c6deaa34e2b9be49f7a9a2c8901, State: Initialized, Role: FOLLOWER
I20261117 09:06:31.493734 22302 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.493750 22303 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.493882 22306 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261117 09:06:31.493896 22302 ts_tablet_manager.cc:1480] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:31.493927 22306 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.493990 22280 heartbeater.cc:500] Master 127.21.116.126:35417 was elected leader, sending a full tablet report...
I20261117 09:06:31.493901 22303 ts_tablet_manager.cc:1480] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:31.494043 22205 heartbeater.cc:500] Master 127.21.116.126:35417 was elected leader, sending a full tablet report...
I20261117 09:06:31.494848 22306 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:31.496824 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" is_pre_election: true
I20261117 09:06:31.496907 22254 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d43c0419874c329f139eea1012c23e in term 0.
I20261117 09:06:31.497098 22144 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901; no voters:
I20261117 09:06:31.497210 22306 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261117 09:06:31.497257 22306 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:31.497279 22306 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:31.497840 22306 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.497941 22306 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 1 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:31.498137 22103 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4" is_pre_election: true
I20261117 09:06:31.498188 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:31.498246 22254 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:31.498245 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
W20261117 09:06:31.498427 22145 leader_election.cc:343] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161): Illegal state: must be running to vote when last-logged opid is not known
I20261117 09:06:31.498503 22301 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap starting.
W20261117 09:06:31.498514 22145 leader_election.cc:343] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161): Illegal state: must be running to vote when last-logged opid is not known
I20261117 09:06:31.498925 22301 tablet_bootstrap.cc:654] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:31.498926 22254 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d43c0419874c329f139eea1012c23e in term 1.
I20261117 09:06:31.499094 22144 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901; no voters: 224ffdac0c8c486d912551a7a37ddfe4
I20261117 09:06:31.499162 22306 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:31.499204 22306 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 1 LEADER]: Becoming Leader. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Running, Role: LEADER
I20261117 09:06:31.499240 22306 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
W20261117 09:06:31.499796 22145 consensus_peers.cc:597] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e -> Peer 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161): Couldn't send request to peer 224ffdac0c8c486d912551a7a37ddfe4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20261117 09:06:31.499738 21998 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d43c0419874c329f139eea1012c23e (127.21.116.66). New cstate: current_term: 1 leader_uuid: "44d43c0419874c329f139eea1012c23e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:31.499974 22308 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261117 09:06:31.502720 22301 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: No bootstrap required, opened a new log
I20261117 09:06:31.502822 22301 ts_tablet_manager.cc:1449] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20261117 09:06:31.503016 22301 raft_consensus.cc:360] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.503129 22301 raft_consensus.cc:386] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:31.503189 22301 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Initialized, Role: FOLLOWER
I20261117 09:06:31.503284 22301 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.503469 22301 ts_tablet_manager.cc:1480] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:31.503507 22129 heartbeater.cc:500] Master 127.21.116.126:35417 was elected leader, sending a full tablet report...
I20261117 09:06:31.516702 21969 tablet.cc:1657] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: MemRowSet was empty: no flush needed.
I20261117 09:06:31.521808 21969 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20261117 09:06:31.526507 22319 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261117 09:06:31.526564 22319 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.526703 22319 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:31.529681 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e" is_pre_election: true
I20261117 09:06:31.530103 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" is_pre_election: true
I20261117 09:06:31.530296 22068 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:31.530448 22319 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20261117 09:06:31.544866 21969 cluster_itest_util.cc:258] Not converged past 27 yet: 1.40 1.40 0.0
I20261117 09:06:31.593786 22102 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 0 FOLLOWER]: Advancing to term 1
W20261117 09:06:31.676709 22306 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20261117 09:06:31.676777 22306 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20261117 09:06:31.777263 22319 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d43c0419874c329f139eea1012c23e)
I20261117 09:06:31.777350 22319 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:31.777948 22319 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.778097 22319 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 2 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:31.778307 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:31.778358 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:31.778601 22068 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:31.778689 22319 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261117 09:06:31.793949 22102 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 2 FOLLOWER]: Rejecting Update request from peer 44d43c0419874c329f139eea1012c23e for earlier term 1. Current term is 2. Ops: []
I20261117 09:06:31.794173 22306 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: INVALID_TERM, Last received: 1.40, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261117 09:06:31.814365 22344 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 1 LEADER]: Stepping down as leader of term 1
I20261117 09:06:31.814435 22344 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Running, Role: LEADER
I20261117 09:06:31.814486 22344 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.814585 22344 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:31.903121 22319 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:31.903214 22319 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 2 FOLLOWER]: Advancing to term 3
I20261117 09:06:31.903842 22319 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.904006 22319 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 3 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:31.904204 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:31.904281 22254 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 1 FOLLOWER]: Advancing to term 3
I20261117 09:06:31.904208 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 3 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:31.904331 22179 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 2 FOLLOWER]: Advancing to term 3
I20261117 09:06:31.904800 22254 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224ffdac0c8c486d912551a7a37ddfe4 in term 3.
I20261117 09:06:31.904978 22179 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224ffdac0c8c486d912551a7a37ddfe4 in term 3.
I20261117 09:06:31.904979 22068 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4, abe59c6deaa34e2b9be49f7a9a2c8901; no voters:
I20261117 09:06:31.905107 22319 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 3 FOLLOWER]: Leader election won for term 3
I20261117 09:06:31.905189 22319 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 3 LEADER]: Becoming Leader. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Running, Role: LEADER
I20261117 09:06:31.905237 22319 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:31.905629 22254 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 3 FOLLOWER]: Refusing update from remote peer 224ffdac0c8c486d912551a7a37ddfe4: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20261117 09:06:31.905629 22179 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 3 FOLLOWER]: Refusing update from remote peer 224ffdac0c8c486d912551a7a37ddfe4: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20261117 09:06:31.905689 21997 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 reported cstate change: term changed from 1 to 3, leader changed from 44d43c0419874c329f139eea1012c23e (127.21.116.66) to 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65). New cstate: current_term: 3 leader_uuid: "224ffdac0c8c486d912551a7a37ddfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:31.905829 22351 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261117 09:06:31.906013 22351 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261117 09:06:32.175280 22307 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 224ffdac0c8c486d912551a7a37ddfe4)
I20261117 09:06:32.175369 22307 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 3 FOLLOWER]: Advancing to term 4
I20261117 09:06:32.176035 22307 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.176221 22307 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 4 election: Requested vote from peers 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:32.179669 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:32.179682 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:32.179924 22220 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abe59c6deaa34e2b9be49f7a9a2c8901; no voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e
I20261117 09:06:32.180032 22307 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20261117 09:06:32.191332 22254 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Rejecting Update request from peer 224ffdac0c8c486d912551a7a37ddfe4 for earlier term 3. Current term is 4. Ops: []
I20261117 09:06:32.191573 22351 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: INVALID_TERM, Last received: 3.41, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261117 09:06:32.208789 22351 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 3 LEADER]: Stepping down as leader of term 3
I20261117 09:06:32.208848 22351 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Running, Role: LEADER
I20261117 09:06:32.208911 22351 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.209013 22351 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 3 FOLLOWER]: Advancing to term 4
I20261117 09:06:32.268577 22344 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 224ffdac0c8c486d912551a7a37ddfe4)
I20261117 09:06:32.268679 22344 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 3 FOLLOWER]: Advancing to term 4
I20261117 09:06:32.269371 22344 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.269531 22344 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 4 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:32.269750 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:32.269750 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:32.269876 22254 raft_consensus.cc:2415] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d43c0419874c329f139eea1012c23e in current term 4: Already voted for candidate abe59c6deaa34e2b9be49f7a9a2c8901 in this term.
I20261117 09:06:32.270368 22102 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d43c0419874c329f139eea1012c23e in term 4.
I20261117 09:06:32.270565 22145 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e; no voters: abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:32.270648 22344 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 4 FOLLOWER]: Leader election won for term 4
I20261117 09:06:32.270709 22344 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 4 LEADER]: Becoming Leader. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Running, Role: LEADER
I20261117 09:06:32.270753 22344 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.271129 22254 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Refusing update from remote peer 44d43c0419874c329f139eea1012c23e: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20261117 09:06:32.271137 22102 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 4 FOLLOWER]: Refusing update from remote peer 44d43c0419874c329f139eea1012c23e: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 4 index: 42. (index mismatch)
I20261117 09:06:32.271193 21999 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e reported cstate change: term changed from 3 to 4, leader changed from 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65) to 44d43c0419874c329f139eea1012c23e (127.21.116.66). New cstate: current_term: 4 leader_uuid: "44d43c0419874c329f139eea1012c23e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:32.271332 22356 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261117 09:06:32.271401 22344 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261117 09:06:32.549862 22351 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d43c0419874c329f139eea1012c23e)
I20261117 09:06:32.549938 22351 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 4 FOLLOWER]: Advancing to term 5
I20261117 09:06:32.550588 22351 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.550740 22351 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 5 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:32.550927 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:32.550923 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:32.551164 22068 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:32.551273 22351 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20261117 09:06:32.552901 22102 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Rejecting Update request from peer 44d43c0419874c329f139eea1012c23e for earlier term 4. Current term is 5. Ops: []
I20261117 09:06:32.553112 22344 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: INVALID_TERM, Last received: 4.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20261117 09:06:32.630430 22307 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d43c0419874c329f139eea1012c23e)
I20261117 09:06:32.630523 22307 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 4 FOLLOWER]: Advancing to term 5
I20261117 09:06:32.631178 22307 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.631335 22307 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 5 election: Requested vote from peers 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:32.631562 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:32.631559 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 5 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:32.631673 22102 raft_consensus.cc:2415] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate abe59c6deaa34e2b9be49f7a9a2c8901 in current term 5: Already voted for candidate 224ffdac0c8c486d912551a7a37ddfe4 in this term.
I20261117 09:06:32.631826 22220 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abe59c6deaa34e2b9be49f7a9a2c8901; no voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e
I20261117 09:06:32.631904 22307 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20261117 09:06:32.635306 22102 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Rejecting Update request from peer 44d43c0419874c329f139eea1012c23e for earlier term 4. Current term is 5. Ops: []
I20261117 09:06:32.635494 22356 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: INVALID_TERM, Last received: 4.42, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20261117 09:06:32.645280 22344 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 4 LEADER]: Stepping down as leader of term 4
I20261117 09:06:32.645324 22344 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Running, Role: LEADER
I20261117 09:06:32.645370 22344 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.645393 22254 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 5 FOLLOWER]: Rejecting Update request from peer 44d43c0419874c329f139eea1012c23e for earlier term 4. Current term is 5. Ops: []
I20261117 09:06:32.645455 22344 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 4 FOLLOWER]: Advancing to term 5
I20261117 09:06:32.665555 22351 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:32.665612 22351 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 5 FOLLOWER]: Advancing to term 6
I20261117 09:06:32.666155 22351 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.666289 22351 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 6 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:32.666484 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 6 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:32.666514 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 6 candidate_status { last_received { term: 4 index: 42 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:32.666540 22179 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 5 FOLLOWER]: Advancing to term 6
I20261117 09:06:32.666579 22254 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 5 FOLLOWER]: Advancing to term 6
I20261117 09:06:32.667061 22179 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224ffdac0c8c486d912551a7a37ddfe4 in term 6.
I20261117 09:06:32.667084 22254 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224ffdac0c8c486d912551a7a37ddfe4 in term 6.
I20261117 09:06:32.667223 22069 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e; no voters:
I20261117 09:06:32.667338 22351 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 6 FOLLOWER]: Leader election won for term 6
I20261117 09:06:32.667397 22351 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 6 LEADER]: Becoming Leader. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Running, Role: LEADER
I20261117 09:06:32.667444 22351 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 4.42, Last appended by leader: 42, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.667793 22179 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 6 FOLLOWER]: Refusing update from remote peer 224ffdac0c8c486d912551a7a37ddfe4: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 6 index: 43. (index mismatch)
I20261117 09:06:32.667793 22254 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 6 FOLLOWER]: Refusing update from remote peer 224ffdac0c8c486d912551a7a37ddfe4: Log matching property violated. Preceding OpId in replica: term: 4 index: 42. Preceding OpId from leader: term: 6 index: 43. (index mismatch)
I20261117 09:06:32.667860 21998 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 reported cstate change: term changed from 4 to 6, leader changed from 44d43c0419874c329f139eea1012c23e (127.21.116.66) to 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65). New cstate: current_term: 6 leader_uuid: "224ffdac0c8c486d912551a7a37ddfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:32.667996 22357 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20261117 09:06:32.668161 22357 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20261117 09:06:32.769189 22307 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 224ffdac0c8c486d912551a7a37ddfe4)
I20261117 09:06:32.769270 22307 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 6 FOLLOWER]: Advancing to term 7
I20261117 09:06:32.769906 22307 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.770069 22307 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 7 election: Requested vote from peers 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:32.770242 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:32.770282 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:32.770504 22220 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abe59c6deaa34e2b9be49f7a9a2c8901; no voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e
I20261117 09:06:32.770601 22307 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20261117 09:06:32.786154 22254 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 7 FOLLOWER]: Rejecting Update request from peer 224ffdac0c8c486d912551a7a37ddfe4 for earlier term 6. Current term is 7. Ops: []
I20261117 09:06:32.786348 22319 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: INVALID_TERM, Last received: 6.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20261117 09:06:32.894583 22254 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 7 FOLLOWER]: Rejecting Update request from peer 224ffdac0c8c486d912551a7a37ddfe4 for earlier term 6. Current term is 7. Ops: []
I20261117 09:06:32.894815 22357 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: INVALID_TERM, Last received: 6.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20261117 09:06:32.917003 22307 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:32.917093 22307 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 7 FOLLOWER]: Advancing to term 8
I20261117 09:06:32.917725 22307 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.917867 22307 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 8 election: Requested vote from peers 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:32.918035 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 8 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:32.918099 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 8 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:32.918237 22220 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abe59c6deaa34e2b9be49f7a9a2c8901; no voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e
I20261117 09:06:32.918341 22307 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20261117 09:06:32.997222 22357 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 6 LEADER]: Stepping down as leader of term 6
I20261117 09:06:32.997316 22357 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Running, Role: LEADER
I20261117 09:06:32.997372 22357 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 6.43, Last appended by leader: 43, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:32.997498 22357 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 6 FOLLOWER]: Advancing to term 7
I20261117 09:06:33.040226 22344 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 224ffdac0c8c486d912551a7a37ddfe4)
I20261117 09:06:33.040321 22344 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 6 FOLLOWER]: Advancing to term 7
I20261117 09:06:33.040952 22344 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.041128 22344 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 7 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:33.041347 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:33.041347 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 7 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:33.041483 22254 raft_consensus.cc:2390] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d43c0419874c329f139eea1012c23e for earlier term 7. Current term is 8.
I20261117 09:06:33.041674 22144 leader_election.cc:400] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 7 election: Vote denied by peer abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241) with higher term. Message: Invalid argument: T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d43c0419874c329f139eea1012c23e for earlier term 7. Current term is 8.
I20261117 09:06:33.041723 22144 leader_election.cc:403] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20261117 09:06:33.041785 22344 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 7 FOLLOWER]: Advancing to term 8
I20261117 09:06:33.042047 22102 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d43c0419874c329f139eea1012c23e in term 7.
I20261117 09:06:33.042322 22344 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241) with higher term. Message: Invalid argument: T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d43c0419874c329f139eea1012c23e for earlier term 7. Current term is 8.
I20261117 09:06:33.074818 22307 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:33.074916 22307 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 8 FOLLOWER]: Advancing to term 9
I20261117 09:06:33.075461 22307 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.075584 22307 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 9 election: Requested vote from peers 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:33.075795 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 9 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:33.075809 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" candidate_term: 9 candidate_status { last_received { term: 6 index: 43 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:33.075870 22102 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 7 FOLLOWER]: Advancing to term 9
I20261117 09:06:33.075897 22179 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 8 FOLLOWER]: Advancing to term 9
I20261117 09:06:33.076386 22102 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe59c6deaa34e2b9be49f7a9a2c8901 in term 9.
I20261117 09:06:33.076414 22179 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe59c6deaa34e2b9be49f7a9a2c8901 in term 9.
I20261117 09:06:33.076574 22220 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901; no voters:
I20261117 09:06:33.076656 22307 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 9 FOLLOWER]: Leader election won for term 9
I20261117 09:06:33.076759 22307 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 9 LEADER]: Becoming Leader. State: Replica: abe59c6deaa34e2b9be49f7a9a2c8901, State: Running, Role: LEADER
I20261117 09:06:33.076798 22307 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 6.43, Last appended by leader: 43, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.077162 22179 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 9 FOLLOWER]: Refusing update from remote peer abe59c6deaa34e2b9be49f7a9a2c8901: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20261117 09:06:33.077243 21997 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 reported cstate change: term changed from 6 to 9, leader changed from 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65) to abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67). New cstate: current_term: 9 leader_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:33.077354 22102 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 9 FOLLOWER]: Refusing update from remote peer abe59c6deaa34e2b9be49f7a9a2c8901: Log matching property violated. Preceding OpId in replica: term: 6 index: 43. Preceding OpId from leader: term: 9 index: 44. (index mismatch)
I20261117 09:06:33.077371 22358 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20261117 09:06:33.077663 22358 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [LEADER]: Connected to new peer: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20261117 09:06:33.178846 22344 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 9 FOLLOWER]: Starting leader election (detected failure of leader abe59c6deaa34e2b9be49f7a9a2c8901)
I20261117 09:06:33.178936 22344 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 9 FOLLOWER]: Advancing to term 10
I20261117 09:06:33.179528 22344 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.179677 22344 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 10 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:33.179899 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:33.179903 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 10 candidate_status { last_received { term: 9 index: 44 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:33.179976 22102 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 9 FOLLOWER]: Advancing to term 10
I20261117 09:06:33.180547 22102 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d43c0419874c329f139eea1012c23e in term 10.
I20261117 09:06:33.180732 22145 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e; no voters: abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:33.180819 22344 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 10 FOLLOWER]: Leader election won for term 10
I20261117 09:06:33.180910 22344 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 10 LEADER]: Becoming Leader. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Running, Role: LEADER
I20261117 09:06:33.180948 22344 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.181313 22254 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 9 LEADER]: Stepping down as leader of term 9
I20261117 09:06:33.181376 22254 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: abe59c6deaa34e2b9be49f7a9a2c8901, State: Running, Role: LEADER
I20261117 09:06:33.181313 22102 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 10 FOLLOWER]: Refusing update from remote peer 44d43c0419874c329f139eea1012c23e: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 10 index: 45. (index mismatch)
I20261117 09:06:33.181345 21999 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e reported cstate change: term changed from 9 to 10, leader changed from abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67) to 44d43c0419874c329f139eea1012c23e (127.21.116.66). New cstate: current_term: 10 leader_uuid: "44d43c0419874c329f139eea1012c23e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:33.181429 22254 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 9.44, Last appended by leader: 44, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.181526 22254 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 9 FOLLOWER]: Advancing to term 10
I20261117 09:06:33.181583 22361 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20261117 09:06:33.182215 22254 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 10 FOLLOWER]: Refusing update from remote peer 44d43c0419874c329f139eea1012c23e: Log matching property violated. Preceding OpId in replica: term: 9 index: 44. Preceding OpId from leader: term: 10 index: 45. (index mismatch)
I20261117 09:06:33.182403 22344 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 44, Time since last communication: 0.000s
I20261117 09:06:33.283113 22357 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 44d43c0419874c329f139eea1012c23e)
I20261117 09:06:33.283195 22357 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 10 FOLLOWER]: Advancing to term 11
I20261117 09:06:33.283829 22357 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.283988 22102 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 11 FOLLOWER]: Rejecting Update request from peer 44d43c0419874c329f139eea1012c23e for earlier term 10. Current term is 11. Ops: []
I20261117 09:06:33.284044 22357 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 11 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:33.284104 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 11 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:33.284148 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 11 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:33.284214 22344 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 }, Status: INVALID_TERM, Last received: 10.45, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20261117 09:06:33.284279 22069 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:33.284343 22357 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20261117 09:06:33.284857 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.286057 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.288242 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.292411 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.300576 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.316737 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.336623 22361 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 10 LEADER]: Stepping down as leader of term 10
I20261117 09:06:33.336673 22361 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 44d43c0419874c329f139eea1012c23e, State: Running, Role: LEADER
I20261117 09:06:33.336725 22361 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 10.45, Last appended by leader: 45, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.336819 22361 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 10 FOLLOWER]: Advancing to term 11
I20261117 09:06:33.348891 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.410764 22357 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:33.410844 22357 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 11 FOLLOWER]: Advancing to term 12
I20261117 09:06:33.411446 22357 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.411594 22357 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 12 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
I20261117 09:06:33.411792 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 12 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:33.411793 22179 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "224ffdac0c8c486d912551a7a37ddfe4" candidate_term: 12 candidate_status { last_received { term: 10 index: 45 } } ignore_live_leader: false dest_uuid: "44d43c0419874c329f139eea1012c23e"
I20261117 09:06:33.411883 22179 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 11 FOLLOWER]: Advancing to term 12
I20261117 09:06:33.412354 22179 raft_consensus.cc:2490] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 224ffdac0c8c486d912551a7a37ddfe4 in term 12.
I20261117 09:06:33.412570 22069 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e; no voters: abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:33.412671 22357 raft_consensus.cc:2826] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 12 FOLLOWER]: Leader election won for term 12
I20261117 09:06:33.412731 22357 raft_consensus.cc:705] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 12 LEADER]: Becoming Leader. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Running, Role: LEADER
I20261117 09:06:33.412782 22357 consensus_queue.cc:238] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 10.45, Last appended by leader: 45, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.413054 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.413127 22254 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 10 FOLLOWER]: Advancing to term 12
I20261117 09:06:33.413213 22179 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 12 FOLLOWER]: Refusing update from remote peer 224ffdac0c8c486d912551a7a37ddfe4: Log matching property violated. Preceding OpId in replica: term: 10 index: 45. Preceding OpId from leader: term: 12 index: 46. (index mismatch)
I20261117 09:06:33.413275 21997 catalog_manager.cc:5840] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 reported cstate change: term changed from 10 to 12, leader changed from 44d43c0419874c329f139eea1012c23e (127.21.116.66) to 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65). New cstate: current_term: 12 leader_uuid: "224ffdac0c8c486d912551a7a37ddfe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:33.413393 22362 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20261117 09:06:33.413825 22254 raft_consensus.cc:1285] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 12 FOLLOWER]: Refusing update from remote peer 224ffdac0c8c486d912551a7a37ddfe4: Log matching property violated. Preceding OpId in replica: term: 10 index: 45. Preceding OpId from leader: term: 12 index: 46. (index mismatch)
I20261117 09:06:33.413987 22362 consensus_queue.cc:1049] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20261117 09:06:33.541244 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:33.541352 21969 log_reader.cc:385] T c62d3ed753144c988a887b52c9eb4491: removed 1 log segments from log reader
I20261117 09:06:33.541378 21969 log.cc:1079] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Deleting log segment in path: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root/wals/c62d3ed753144c988a887b52c9eb4491/wal-000000001 (ops 1-40)
I20261117 09:06:33.541815 21969 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20261117 09:06:33.541893 21969 master.cc:585] Master@127.21.116.126:35417 shutting down...
I20261117 09:06:33.614266 21969 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 1 LEADER]: Raft consensus shutting down.
I20261117 09:06:33.614372 21969 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:33.614427 21969 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1c1609560ec2400ca45db3811b388372: stopping tablet replica
I20261117 09:06:33.615202 22361 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 224ffdac0c8c486d912551a7a37ddfe4)
I20261117 09:06:33.615259 22361 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 12 FOLLOWER]: Advancing to term 13
I20261117 09:06:33.616245 22361 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.616384 22361 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 13 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161)
I20261117 09:06:33.616526 22254 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 13 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901"
I20261117 09:06:33.616526 22102 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "c62d3ed753144c988a887b52c9eb4491" candidate_uuid: "44d43c0419874c329f139eea1012c23e" candidate_term: 13 candidate_status { last_received { term: 12 index: 46 } } ignore_live_leader: false dest_uuid: "224ffdac0c8c486d912551a7a37ddfe4"
I20261117 09:06:33.616813 22145 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d43c0419874c329f139eea1012c23e; no voters: 224ffdac0c8c486d912551a7a37ddfe4, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:33.616892 22361 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20261117 09:06:33.626644 21969 master.cc:607] Master@127.21.116.126:35417 shutdown complete.
I20261117 09:06:33.627918 21969 tablet_server.cc:179] TabletServer@127.21.116.65:0 shutting down...
I20261117 09:06:33.628615 22179 raft_consensus.cc:1250] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 13 FOLLOWER]: Rejecting Update request from peer 224ffdac0c8c486d912551a7a37ddfe4 for earlier term 12. Current term is 13. Ops: []
I20261117 09:06:33.628857 22362 consensus_queue.cc:1060] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 }, Status: INVALID_TERM, Last received: 12.46, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20261117 09:06:33.628957 22362 raft_consensus.cc:3091] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 12 LEADER]: Stepping down as leader of term 12
I20261117 09:06:33.628978 22362 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Running, Role: LEADER
I20261117 09:06:33.629031 22362 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.629107 22362 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 12 FOLLOWER]: Advancing to term 13
I20261117 09:06:33.680032 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:33.680233 21969 tablet_replica.cc:333] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: stopping tablet replica
I20261117 09:06:33.680286 21969 raft_consensus.cc:2261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 13 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:33.680329 21969 raft_consensus.cc:2290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 13 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:33.692111 21969 tablet_server.cc:196] TabletServer@127.21.116.65:0 shutdown complete.
I20261117 09:06:33.693231 21969 tablet_server.cc:179] TabletServer@127.21.116.66:0 shutting down...
I20261117 09:06:33.695595 22358 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 224ffdac0c8c486d912551a7a37ddfe4)
I20261117 09:06:33.695657 22358 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 12 FOLLOWER]: Advancing to term 13
I20261117 09:06:33.696282 22358 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.696401 22358 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 13 election: Requested vote from peers 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
W20261117 09:06:33.696703 22220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.116.65:33161: connect: Connection refused (error 111)
W20261117 09:06:33.697075 22220 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 224ffdac0c8c486d912551a7a37ddfe4 (127.21.116.65:33161): Network error: Client connection negotiation failed: client connection to 127.21.116.65:33161: connect: Connection refused (error 111)
W20261117 09:06:33.697147 22220 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20261117 09:06:33.697181 22220 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abe59c6deaa34e2b9be49f7a9a2c8901; no voters: 224ffdac0c8c486d912551a7a37ddfe4, 44d43c0419874c329f139eea1012c23e
I20261117 09:06:33.697234 22358 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20261117 09:06:33.700052 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:33.700133 21969 tablet_replica.cc:333] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e: stopping tablet replica
I20261117 09:06:33.700203 21969 raft_consensus.cc:2261] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 13 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:33.700243 21969 raft_consensus.cc:2290] T c62d3ed753144c988a887b52c9eb4491 P 44d43c0419874c329f139eea1012c23e [term 13 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:33.701526 21969 tablet_server.cc:196] TabletServer@127.21.116.66:0 shutdown complete.
I20261117 09:06:33.702203 21969 tablet_server.cc:179] TabletServer@127.21.116.67:0 shutting down...
I20261117 09:06:33.715976 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:33.716069 21969 tablet_replica.cc:333] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901: stopping tablet replica
I20261117 09:06:33.716118 21969 raft_consensus.cc:2261] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 13 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:33.716159 21969 raft_consensus.cc:2290] T c62d3ed753144c988a887b52c9eb4491 P abe59c6deaa34e2b9be49f7a9a2c8901 [term 13 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:33.727519 21969 tablet_server.cc:196] TabletServer@127.21.116.67:0 shutdown complete.
I20261117 09:06:33.728125 21969 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20261117 09:06:33.728261 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:33.728358 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:33.729317 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:33.729360 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:33.729470 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:33.729568 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:33.729651 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:33.729681 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:33.729692 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906393729692 us; error 0 us; skew 500 ppm
I20261117 09:06:33.730165 21969 webserver.cc:577] Webserver started at http://127.21.116.65:44771/ using document root <none> and password file <none>
I20261117 09:06:33.730224 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:33.730255 21969 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261117 09:06:33.730861 21969 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:33.732007 22377 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:33.732123 21969 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:33.732172 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "224ffdac0c8c486d912551a7a37ddfe4"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:33.732218 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5758
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:33.742794 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:33.742928 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:33.743041 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:33.743283 22384 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261117 09:06:33.743820 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261117 09:06:33.743855 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:33.743873 21969 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261117 09:06:33.744217 21969 ts_tablet_manager.cc:662] Registered 1 tablets
I20261117 09:06:33.744244 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:33.744280 22384 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap starting.
I20261117 09:06:33.745597 22384 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261117 09:06:33.745841 22384 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap complete.
I20261117 09:06:33.745934 22384 ts_tablet_manager.cc:1449] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20261117 09:06:33.746047 22384 raft_consensus.cc:360] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 13 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.746124 22384 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Initialized, Role: FOLLOWER
I20261117 09:06:33.746169 22384 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.746310 22384 ts_tablet_manager.cc:1480] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:33.746924 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:33161
I20261117 09:06:33.749632 22443 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:33161 every 8 connection(s)
W20261117 09:06:33.750028 22444 heartbeater.cc:648] Failed to heartbeat to 127.21.116.126:35417 (0 consecutive failures): Network error: Failed to ping master at 127.21.116.126:35417: Client connection negotiation failed: client connection to 127.21.116.126:35417: connect: Connection refused (error 111)
W20261117 09:06:33.750466 22444 heartbeater.cc:413] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20261117 09:06:33.752974 21969 tablet.cc:2141] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20261117 09:06:33.753820 21969 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20261117 09:06:33.753875 21969 tablet_server.cc:179] TabletServer@127.21.116.65:33161 shutting down...
I20261117 09:06:33.840867 22430 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:33.840969 22430 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 13 FOLLOWER]: Advancing to term 14
I20261117 09:06:33.841753 22430 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.841961 22430 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 14 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
W20261117 09:06:33.842592 22381 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917): Network error: Client connection negotiation failed: client connection to 127.21.116.66:41917: connect: Connection refused (error 111)
W20261117 09:06:33.842690 22380 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241): Network error: Client connection negotiation failed: client connection to 127.21.116.67:40241: connect: Connection refused (error 111)
I20261117 09:06:33.842734 22380 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:33.842800 22430 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20261117 09:06:33.970278 22430 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:33.970369 22430 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 14 FOLLOWER]: Advancing to term 15
I20261117 09:06:33.971005 22430 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:33.971164 22430 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 15 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
W20261117 09:06:33.971783 22381 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917): Network error: Client connection negotiation failed: client connection to 127.21.116.66:41917: connect: Connection refused (error 111)
W20261117 09:06:33.971860 22380 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241): Network error: Client connection negotiation failed: client connection to 127.21.116.67:40241: connect: Connection refused (error 111)
I20261117 09:06:33.971920 22380 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:33.971992 22430 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20261117 09:06:34.000007 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:34.000155 21969 tablet_replica.cc:333] stopping tablet replica
I20261117 09:06:34.000247 21969 raft_consensus.cc:2261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:34.000298 21969 raft_consensus.cc:2290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:34.011893 21969 tablet_server.cc:196] TabletServer@127.21.116.65:33161 shutdown complete.
I20261117 09:06:34.012960 21969 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20261117 09:06:34.013079 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:34.013172 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:34.014187 22457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.014216 22455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.014307 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:34.014322 22454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.014561 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:34.014599 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:34.014611 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906394014610 us; error 0 us; skew 500 ppm
I20261117 09:06:34.015118 21969 webserver.cc:577] Webserver started at http://127.21.116.65:44771/ using document root <none> and password file <none>
I20261117 09:06:34.015196 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:34.015230 21969 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261117 09:06:34.015767 21969 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.016829 22464 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.016937 21969 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:34.016983 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "224ffdac0c8c486d912551a7a37ddfe4"
format_stamp: "Formatted at 2026-11-17 09:06:31 on dist-test-slave-1b7w"
I20261117 09:06:34.017031 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5758
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:34.028618 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:34.028765 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:34.028879 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:34.029546 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261117 09:06:34.029597 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:34.029982 21969 ts_tablet_manager.cc:662] Registered 1 tablets
I20261117 09:06:34.030009 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.030019 22471 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap starting.
I20261117 09:06:34.031157 22471 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261117 09:06:34.031340 22471 tablet_bootstrap.cc:492] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Bootstrap complete.
I20261117 09:06:34.031422 22471 ts_tablet_manager.cc:1449] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:34.031526 22471 raft_consensus.cc:360] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:34.031582 22471 raft_consensus.cc:748] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224ffdac0c8c486d912551a7a37ddfe4, State: Initialized, Role: FOLLOWER
I20261117 09:06:34.031628 22471 consensus_queue.cc:261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 46, Last appended: 12.46, Last appended by leader: 46, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:34.031787 22471 ts_tablet_manager.cc:1480] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20261117 09:06:34.032941 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:33161
I20261117 09:06:34.033360 22530 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:33161 every 8 connection(s)
W20261117 09:06:34.034371 22531 heartbeater.cc:413] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20261117 09:06:34.036705 21969 tablet.cc:2141] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/timestamp_advancement-itest.cc:362: Failure
Value of: s.IsCorruption()
Actual: false
Expected: true
OK
I20261117 09:06:34.037531 21969 tablet_server.cc:179] TabletServer@127.21.116.65:33161 shutting down...
I20261117 09:06:34.152096 22504 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:34.152179 22504 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 15 FOLLOWER]: Advancing to term 16
I20261117 09:06:34.152818 22504 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:34.153002 22504 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 16 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
W20261117 09:06:34.153622 22467 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241): Network error: Client connection negotiation failed: client connection to 127.21.116.67:40241: connect: Connection refused (error 111)
W20261117 09:06:34.153715 22468 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917): Network error: Client connection negotiation failed: client connection to 127.21.116.66:41917: connect: Connection refused (error 111)
I20261117 09:06:34.153775 22468 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:34.153834 22504 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20261117 09:06:34.264328 22504 raft_consensus.cc:499] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:34.264431 22504 raft_consensus.cc:3096] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 16 FOLLOWER]: Advancing to term 17
I20261117 09:06:34.265106 22504 raft_consensus.cc:521] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abe59c6deaa34e2b9be49f7a9a2c8901" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 40241 } } peers { permanent_uuid: "224ffdac0c8c486d912551a7a37ddfe4" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 33161 } } peers { permanent_uuid: "44d43c0419874c329f139eea1012c23e" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 41917 } }
I20261117 09:06:34.265270 22504 leader_election.cc:290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 17 election: Requested vote from peers abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241), 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917)
W20261117 09:06:34.265863 22467 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer abe59c6deaa34e2b9be49f7a9a2c8901 (127.21.116.67:40241): Network error: Client connection negotiation failed: client connection to 127.21.116.67:40241: connect: Connection refused (error 111)
W20261117 09:06:34.265925 22468 leader_election.cc:336] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 44d43c0419874c329f139eea1012c23e (127.21.116.66:41917): Network error: Client connection negotiation failed: client connection to 127.21.116.66:41917: connect: Connection refused (error 111)
I20261117 09:06:34.265981 22468 leader_election.cc:304] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 224ffdac0c8c486d912551a7a37ddfe4; no voters: 44d43c0419874c329f139eea1012c23e, abe59c6deaa34e2b9be49f7a9a2c8901
I20261117 09:06:34.266048 22504 raft_consensus.cc:2771] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20261117 09:06:34.283738 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:34.283886 21969 tablet_replica.cc:333] stopping tablet replica
I20261117 09:06:34.283948 21969 raft_consensus.cc:2261] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 17 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:34.284006 21969 raft_consensus.cc:2290] T c62d3ed753144c988a887b52c9eb4491 P 224ffdac0c8c486d912551a7a37ddfe4 [term 17 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:34.295574 21969 tablet_server.cc:196] TabletServer@127.21.116.65:33161 shutdown complete.
I20261117 09:06:34.297349 21969 test_util.cc:182] -----------------------------------------------
I20261117 09:06:34.297395 21969 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData.1794906391332839-21969-0
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData (2960 ms)
[ RUN ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap
I20261117 09:06:34.297834 21969 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.116.126:42017
I20261117 09:06:34.297969 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:34.299019 22544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.299113 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.299275 22542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.299319 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:34.299391 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:34.299419 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:34.299435 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906394299436 us; error 0 us; skew 500 ppm
I20261117 09:06:34.299877 21969 webserver.cc:577] Webserver started at http://127.21.116.126:42475/ using document root <none> and password file <none>
I20261117 09:06:34.299948 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:34.299990 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:34.300040 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:34.300292 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/master-0-root/instance:
uuid: "70c99515af434a8f8995cc082ed2a9a5"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.301087 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:34.301523 22549 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.301659 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.301697 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/master-0-root
uuid: "70c99515af434a8f8995cc082ed2a9a5"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.301735 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:34.308830 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:34.308966 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:34.311650 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.126:42017
I20261117 09:06:34.311954 22601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.126:42017 every 8 connection(s)
I20261117 09:06:34.312090 22602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:34.312959 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5: Bootstrap starting.
I20261117 09:06:34.313207 22602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:34.313634 22602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5: No bootstrap required, opened a new log
I20261117 09:06:34.313735 22602 raft_consensus.cc:360] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c99515af434a8f8995cc082ed2a9a5" member_type: VOTER }
I20261117 09:06:34.313786 22602 raft_consensus.cc:386] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:34.313807 22602 raft_consensus.cc:748] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c99515af434a8f8995cc082ed2a9a5, State: Initialized, Role: FOLLOWER
I20261117 09:06:34.313851 22602 consensus_queue.cc:261] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c99515af434a8f8995cc082ed2a9a5" member_type: VOTER }
I20261117 09:06:34.313885 22602 raft_consensus.cc:400] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:34.313910 22602 raft_consensus.cc:499] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:34.313934 22602 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:34.314462 22602 raft_consensus.cc:521] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c99515af434a8f8995cc082ed2a9a5" member_type: VOTER }
I20261117 09:06:34.314522 22602 leader_election.cc:304] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70c99515af434a8f8995cc082ed2a9a5; no voters:
I20261117 09:06:34.314616 22602 leader_election.cc:290] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261117 09:06:34.314646 22605 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:34.314741 22602 sys_catalog.cc:565] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [sys.catalog]: configured and running, proceeding with master startup.
I20261117 09:06:34.314764 22605 raft_consensus.cc:705] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 1 LEADER]: Becoming Leader. State: Replica: 70c99515af434a8f8995cc082ed2a9a5, State: Running, Role: LEADER
I20261117 09:06:34.314810 22605 consensus_queue.cc:238] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c99515af434a8f8995cc082ed2a9a5" member_type: VOTER }
I20261117 09:06:34.315022 22607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70c99515af434a8f8995cc082ed2a9a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c99515af434a8f8995cc082ed2a9a5" member_type: VOTER } }
I20261117 09:06:34.315026 22606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70c99515af434a8f8995cc082ed2a9a5. Latest consensus state: current_term: 1 leader_uuid: "70c99515af434a8f8995cc082ed2a9a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c99515af434a8f8995cc082ed2a9a5" member_type: VOTER } }
I20261117 09:06:34.315152 22606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:34.315119 22607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:34.315380 22611 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261117 09:06:34.315532 22611 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261117 09:06:34.316023 22611 catalog_manager.cc:1398] Generated new cluster ID: 16ff3c3908a24a81aeb7659e4c1447fd
I20261117 09:06:34.316052 21969 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261117 09:06:34.316066 22611 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261117 09:06:34.326650 22611 catalog_manager.cc:1421] Generated new certificate authority record
I20261117 09:06:34.327025 22611 catalog_manager.cc:1555] Loading token signing keys...
I20261117 09:06:34.330264 22611 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5: Generated new TSK 0
I20261117 09:06:34.330332 22611 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261117 09:06:34.331545 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:34.331627 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:34.332456 22623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.332511 22624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.332542 22626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.332713 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:34.332789 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:34.332818 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:34.332839 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906394332840 us; error 0 us; skew 500 ppm
I20261117 09:06:34.333320 21969 webserver.cc:577] Webserver started at http://127.21.116.65:35687/ using document root <none> and password file <none>
I20261117 09:06:34.333384 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:34.333422 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:34.333463 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:34.333662 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root/instance:
uuid: "a553d253650a4085a021354456c9d66a"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.334364 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:34.334781 22631 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.334872 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.334913 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "a553d253650a4085a021354456c9d66a"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.334949 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:34.344235 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:34.344376 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:34.344486 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:34.344628 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:34.344650 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.344677 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:34.344699 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.347224 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:37587
I20261117 09:06:34.347252 22694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:37587 every 8 connection(s)
I20261117 09:06:34.347500 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:34.347585 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:34.348428 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.348460 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.348683 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.348701 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:34.348801 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:34.348842 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:34.348860 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906394348859 us; error 0 us; skew 500 ppm
I20261117 09:06:34.349305 21969 webserver.cc:577] Webserver started at http://127.21.116.66:42899/ using document root <none> and password file <none>
I20261117 09:06:34.349373 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:34.349409 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:34.349448 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:34.349664 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-1-root/instance:
uuid: "680033ff21ed4734971b949bf68b1895"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.350558 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:34.350896 22695 heartbeater.cc:345] Connected to a master server at 127.21.116.126:42017
I20261117 09:06:34.350950 22695 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:34.351013 22707 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.351048 22695 heartbeater.cc:508] Master 127.21.116.126:42017 requested a full tablet report, sending...
I20261117 09:06:34.351181 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.351221 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-1-root
uuid: "680033ff21ed4734971b949bf68b1895"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.351253 22566 ts_manager.cc:210] Registered new tserver with Master: a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:34.351256 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:34.351758 22566 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47124
I20261117 09:06:34.370712 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:34.370888 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:34.371024 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:34.371181 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:34.371206 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.371229 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:34.371250 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.374114 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.66:39487
I20261117 09:06:34.374166 22770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.66:39487 every 8 connection(s)
I20261117 09:06:34.374433 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:34.374528 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:34.375422 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.375629 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:34.375875 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:34.375633 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:34.376865 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:34.376912 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:34.376929 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906394376930 us; error 0 us; skew 500 ppm
I20261117 09:06:34.377389 22771 heartbeater.cc:345] Connected to a master server at 127.21.116.126:42017
I20261117 09:06:34.377444 22771 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:34.377458 21969 webserver.cc:577] Webserver started at http://127.21.116.67:41253/ using document root <none> and password file <none>
I20261117 09:06:34.377535 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:34.377537 22771 heartbeater.cc:508] Master 127.21.116.126:42017 requested a full tablet report, sending...
I20261117 09:06:34.377570 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:34.377616 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:34.377789 22566 ts_manager.cc:210] Registered new tserver with Master: 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487)
I20261117 09:06:34.377848 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-2-root/instance:
uuid: "97a11483cdc54c77975eedc94efd2768"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.378127 22566 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47128
I20261117 09:06:34.378680 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:34.379051 22782 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.379148 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.379187 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-2-root
uuid: "97a11483cdc54c77975eedc94efd2768"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:34.379226 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:34.401938 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:34.402136 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:34.402268 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:34.402441 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:34.402470 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.402494 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:34.402514 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.405400 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.67:43087
I20261117 09:06:34.405459 22845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.67:43087 every 8 connection(s)
I20261117 09:06:34.408020 22846 heartbeater.cc:345] Connected to a master server at 127.21.116.126:42017
I20261117 09:06:34.408070 22846 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:34.408160 22846 heartbeater.cc:508] Master 127.21.116.126:42017 requested a full tablet report, sending...
I20261117 09:06:34.408349 22566 ts_manager.cc:210] Registered new tserver with Master: 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:34.408674 22566 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47132
I20261117 09:06:34.408854 21969 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003278155s
I20261117 09:06:34.418615 22566 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47152:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261117 09:06:34.418732 22566 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261117 09:06:34.422837 22659 tablet_service.cc:1500] Processing CreateTablet for tablet 1e3b46374d4047b8a68c84fda3badbc7 (DEFAULT_TABLE table=test-workload [id=41b121ae7bad4341af8b3bfb74815c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:34.423467 22659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3b46374d4047b8a68c84fda3badbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:34.423511 22735 tablet_service.cc:1500] Processing CreateTablet for tablet 1e3b46374d4047b8a68c84fda3badbc7 (DEFAULT_TABLE table=test-workload [id=41b121ae7bad4341af8b3bfb74815c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:34.423569 22810 tablet_service.cc:1500] Processing CreateTablet for tablet 1e3b46374d4047b8a68c84fda3badbc7 (DEFAULT_TABLE table=test-workload [id=41b121ae7bad4341af8b3bfb74815c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:34.423583 22735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3b46374d4047b8a68c84fda3badbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:34.423661 22810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3b46374d4047b8a68c84fda3badbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:34.424715 22867 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768: Bootstrap starting.
I20261117 09:06:34.424968 22868 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Bootstrap starting.
I20261117 09:06:34.425149 22867 tablet_bootstrap.cc:654] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:34.425354 22868 tablet_bootstrap.cc:654] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:34.425686 22869 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895: Bootstrap starting.
I20261117 09:06:34.425758 22867 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768: No bootstrap required, opened a new log
I20261117 09:06:34.425799 22867 ts_tablet_manager.cc:1449] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:34.425947 22867 raft_consensus.cc:360] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.426005 22867 raft_consensus.cc:386] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:34.426028 22867 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Initialized, Role: FOLLOWER
I20261117 09:06:34.426075 22867 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.426133 22869 tablet_bootstrap.cc:654] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:34.426246 22867 ts_tablet_manager.cc:1480] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20261117 09:06:34.426271 22846 heartbeater.cc:500] Master 127.21.116.126:42017 was elected leader, sending a full tablet report...
I20261117 09:06:34.426592 22868 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: No bootstrap required, opened a new log
I20261117 09:06:34.426641 22868 ts_tablet_manager.cc:1449] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261117 09:06:34.426776 22868 raft_consensus.cc:360] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.426834 22868 raft_consensus.cc:386] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:34.426860 22868 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a553d253650a4085a021354456c9d66a, State: Initialized, Role: FOLLOWER
I20261117 09:06:34.426911 22868 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.427057 22868 ts_tablet_manager.cc:1480] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.427086 22695 heartbeater.cc:500] Master 127.21.116.126:42017 was elected leader, sending a full tablet report...
I20261117 09:06:34.427215 22869 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895: No bootstrap required, opened a new log
I20261117 09:06:34.427256 22869 ts_tablet_manager.cc:1449] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261117 09:06:34.427384 22869 raft_consensus.cc:360] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.427440 22869 raft_consensus.cc:386] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:34.427457 22869 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Initialized, Role: FOLLOWER
I20261117 09:06:34.427508 22869 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.427651 22869 ts_tablet_manager.cc:1480] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:34.427721 22771 heartbeater.cc:500] Master 127.21.116.126:42017 was elected leader, sending a full tablet report...
I20261117 09:06:34.429220 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261117 09:06:34.429281 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.429419 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:34.431668 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895" is_pre_election: true
I20261117 09:06:34.431732 22745 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 0.
I20261117 09:06:34.431771 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a" is_pre_election: true
I20261117 09:06:34.431845 22669 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 0.
I20261117 09:06:34.431861 22785 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters:
I20261117 09:06:34.431927 22872 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261117 09:06:34.431963 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:34.431981 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:34.432449 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.432555 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 1 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:34.432700 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:34.432708 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:34.432786 22745 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:34.432792 22669 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:34.433408 22745 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 1.
I20261117 09:06:34.433408 22669 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 1.
I20261117 09:06:34.433583 22785 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters:
I20261117 09:06:34.433650 22872 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:34.433768 22872 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 1 LEADER]: Becoming Leader. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:34.433812 22872 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.434255 22566 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97a11483cdc54c77975eedc94efd2768 (127.21.116.67). New cstate: current_term: 1 leader_uuid: "97a11483cdc54c77975eedc94efd2768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:34.434324 22877 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261117 09:06:34.434573 22877 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261117 09:06:34.728668 22874 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:34.728756 22874 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:34.729411 22874 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.729605 22874 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 2 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:34.732112 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:34.732139 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:34.732350 22634 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:34.732445 22874 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261117 09:06:34.735503 22669 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Rejecting Update request from peer 97a11483cdc54c77975eedc94efd2768 for earlier term 1. Current term is 2. Ops: []
I20261117 09:06:34.735687 22872 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: INVALID_TERM, Last received: 1.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20261117 09:06:34.769899 22872 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 1 LEADER]: Stepping down as leader of term 1
I20261117 09:06:34.769958 22872 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:34.770025 22872 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.770113 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:34.776305 22875 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:34.776363 22875 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:34.776932 22875 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.777100 22875 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 2 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:34.780005 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:34.780014 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:34.780081 22669 raft_consensus.cc:2415] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 680033ff21ed4734971b949bf68b1895 in current term 2: Already voted for candidate a553d253650a4085a021354456c9d66a in this term.
I20261117 09:06:34.780637 22820 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680033ff21ed4734971b949bf68b1895 in term 2.
I20261117 09:06:34.780818 22709 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters: a553d253650a4085a021354456c9d66a
I20261117 09:06:34.780910 22875 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 2 FOLLOWER]: Leader election won for term 2
I20261117 09:06:34.781008 22875 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 2 LEADER]: Becoming Leader. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:34.781056 22875 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:34.781420 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 2 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20261117 09:06:34.781425 22669 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20261117 09:06:34.781613 22903 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20261117 09:06:34.781589 22563 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 reported cstate change: term changed from 1 to 2, leader changed from 97a11483cdc54c77975eedc94efd2768 (127.21.116.67) to 680033ff21ed4734971b949bf68b1895 (127.21.116.66). New cstate: current_term: 2 leader_uuid: "680033ff21ed4734971b949bf68b1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:34.781692 22875 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20261117 09:06:35.063395 22874 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 680033ff21ed4734971b949bf68b1895)
I20261117 09:06:35.063493 22874 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 2 FOLLOWER]: Advancing to term 3
I20261117 09:06:35.064141 22874 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.064311 22874 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 3 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:35.064538 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:35.064543 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:35.064790 22633 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:35.064895 22874 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261117 09:06:35.071789 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 680033ff21ed4734971b949bf68b1895)
I20261117 09:06:35.071839 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 2 FOLLOWER]: Advancing to term 3
I20261117 09:06:35.072407 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.072526 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 3 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:35.072675 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:35.072682 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:35.072770 22669 raft_consensus.cc:2415] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 97a11483cdc54c77975eedc94efd2768 in current term 3: Already voted for candidate a553d253650a4085a021354456c9d66a in this term.
I20261117 09:06:35.072906 22783 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768; no voters: 680033ff21ed4734971b949bf68b1895, a553d253650a4085a021354456c9d66a
I20261117 09:06:35.073006 22872 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20261117 09:06:35.079877 22669 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 3 FOLLOWER]: Rejecting Update request from peer 680033ff21ed4734971b949bf68b1895 for earlier term 2. Current term is 3. Ops: []
I20261117 09:06:35.080056 22902 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: INVALID_TERM, Last received: 2.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20261117 09:06:35.092396 22820 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 3 FOLLOWER]: Rejecting Update request from peer 680033ff21ed4734971b949bf68b1895 for earlier term 2. Current term is 3. Ops: []
I20261117 09:06:35.092566 22902 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: INVALID_TERM, Last received: 2.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20261117 09:06:35.158250 22903 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 2 LEADER]: Stepping down as leader of term 2
I20261117 09:06:35.158336 22903 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:35.158430 22903 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.158530 22903 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 2 FOLLOWER]: Advancing to term 3
I20261117 09:06:35.172235 22874 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:35.172298 22874 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 3 FOLLOWER]: Advancing to term 4
I20261117 09:06:35.172853 22874 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.172993 22874 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 4 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:35.173218 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:35.173218 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 4 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:35.173331 22820 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 3 FOLLOWER]: Advancing to term 4
I20261117 09:06:35.173338 22745 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 3 FOLLOWER]: Advancing to term 4
I20261117 09:06:35.173990 22745 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a553d253650a4085a021354456c9d66a in term 4.
I20261117 09:06:35.173990 22820 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a553d253650a4085a021354456c9d66a in term 4.
I20261117 09:06:35.174180 22633 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768, a553d253650a4085a021354456c9d66a; no voters:
I20261117 09:06:35.174301 22874 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 4 FOLLOWER]: Leader election won for term 4
I20261117 09:06:35.174438 22874 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 4 LEADER]: Becoming Leader. State: Replica: a553d253650a4085a021354456c9d66a, State: Running, Role: LEADER
I20261117 09:06:35.174495 22874 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.174881 22745 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 4 FOLLOWER]: Refusing update from remote peer a553d253650a4085a021354456c9d66a: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20261117 09:06:35.174896 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 4 FOLLOWER]: Refusing update from remote peer a553d253650a4085a021354456c9d66a: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 4 index: 17. (index mismatch)
I20261117 09:06:35.174911 22565 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a reported cstate change: term changed from 2 to 4, leader changed from 680033ff21ed4734971b949bf68b1895 (127.21.116.66) to a553d253650a4085a021354456c9d66a (127.21.116.65). New cstate: current_term: 4 leader_uuid: "a553d253650a4085a021354456c9d66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:35.175065 22905 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20261117 09:06:35.175199 22904 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20261117 09:06:35.477340 22903 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a553d253650a4085a021354456c9d66a)
I20261117 09:06:35.477437 22903 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 4 FOLLOWER]: Advancing to term 5
I20261117 09:06:35.478091 22903 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.478267 22903 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 5 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:35.478495 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:35.478505 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:35.478735 22709 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 680033ff21ed4734971b949bf68b1895; no voters: 97a11483cdc54c77975eedc94efd2768, a553d253650a4085a021354456c9d66a
I20261117 09:06:35.478816 22903 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20261117 09:06:35.490438 22745 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Rejecting Update request from peer a553d253650a4085a021354456c9d66a for earlier term 4. Current term is 5. Ops: []
I20261117 09:06:35.490633 22874 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: INVALID_TERM, Last received: 4.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20261117 09:06:35.510823 22874 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 4 LEADER]: Stepping down as leader of term 4
I20261117 09:06:35.510879 22874 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a553d253650a4085a021354456c9d66a, State: Running, Role: LEADER
I20261117 09:06:35.510946 22874 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.511029 22874 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 4 FOLLOWER]: Advancing to term 5
I20261117 09:06:35.520893 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a553d253650a4085a021354456c9d66a)
I20261117 09:06:35.520951 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 4 FOLLOWER]: Advancing to term 5
I20261117 09:06:35.521483 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.521613 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 5 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:35.521799 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:35.521806 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 5 candidate_status { last_received { term: 4 index: 17 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:35.521870 22745 raft_consensus.cc:2415] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 97a11483cdc54c77975eedc94efd2768 in current term 5: Already voted for candidate 680033ff21ed4734971b949bf68b1895 in this term.
I20261117 09:06:35.522459 22669 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 5.
I20261117 09:06:35.522653 22783 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768, a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895
I20261117 09:06:35.522735 22872 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 5 FOLLOWER]: Leader election won for term 5
I20261117 09:06:35.522827 22872 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 5 LEADER]: Becoming Leader. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:35.522868 22872 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 4.17, Last appended by leader: 17, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.523247 22745 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Refusing update from remote peer 97a11483cdc54c77975eedc94efd2768: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20261117 09:06:35.523253 22669 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 5 FOLLOWER]: Refusing update from remote peer 97a11483cdc54c77975eedc94efd2768: Log matching property violated. Preceding OpId in replica: term: 4 index: 17. Preceding OpId from leader: term: 5 index: 18. (index mismatch)
I20261117 09:06:35.523279 22566 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 reported cstate change: term changed from 4 to 5, leader changed from a553d253650a4085a021354456c9d66a (127.21.116.65) to 97a11483cdc54c77975eedc94efd2768 (127.21.116.67). New cstate: current_term: 5 leader_uuid: "97a11483cdc54c77975eedc94efd2768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:35.523432 22907 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20261117 09:06:35.523490 22872 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20261117 09:06:35.625234 22903 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:35.625313 22903 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 5 FOLLOWER]: Advancing to term 6
I20261117 09:06:35.625905 22903 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.626052 22903 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 6 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:35.626197 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:35.626256 22669 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 5 FOLLOWER]: Advancing to term 6
I20261117 09:06:35.626230 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 6 candidate_status { last_received { term: 5 index: 18 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:35.626808 22669 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680033ff21ed4734971b949bf68b1895 in term 6.
I20261117 09:06:35.626976 22708 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, a553d253650a4085a021354456c9d66a; no voters: 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:35.627056 22903 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 6 FOLLOWER]: Leader election won for term 6
I20261117 09:06:35.627101 22903 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 6 LEADER]: Becoming Leader. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:35.627146 22903 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.627542 22669 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 6 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 6 index: 19. (index mismatch)
I20261117 09:06:35.627537 22820 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 5 LEADER]: Stepping down as leader of term 5
I20261117 09:06:35.627516 22565 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 reported cstate change: term changed from 5 to 6, leader changed from 97a11483cdc54c77975eedc94efd2768 (127.21.116.67) to 680033ff21ed4734971b949bf68b1895 (127.21.116.66). New cstate: current_term: 6 leader_uuid: "680033ff21ed4734971b949bf68b1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:35.627616 22820 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:35.627698 22820 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 5.18, Last appended by leader: 18, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.627725 22903 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20261117 09:06:35.627772 22820 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 5 FOLLOWER]: Advancing to term 6
I20261117 09:06:35.628418 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 6 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 5 index: 18. Preceding OpId from leader: term: 6 index: 19. (index mismatch)
I20261117 09:06:35.628712 22903 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20261117 09:06:35.736943 22874 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 6 FOLLOWER]: Starting leader election (detected failure of leader 680033ff21ed4734971b949bf68b1895)
I20261117 09:06:35.737038 22874 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 6 FOLLOWER]: Advancing to term 7
I20261117 09:06:35.737653 22874 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.737820 22874 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 7 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:35.737996 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:35.738001 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:35.738215 22633 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:35.738324 22874 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20261117 09:06:35.743520 22669 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Rejecting Update request from peer 680033ff21ed4734971b949bf68b1895 for earlier term 6. Current term is 7. Ops: []
I20261117 09:06:35.743825 22875 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: INVALID_TERM, Last received: 6.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20261117 09:06:35.813038 22903 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 6 LEADER]: Stepping down as leader of term 6
I20261117 09:06:35.813129 22903 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:35.813217 22903 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 6.19, Last appended by leader: 19, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.813323 22903 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 6 FOLLOWER]: Advancing to term 7
I20261117 09:06:35.854213 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 680033ff21ed4734971b949bf68b1895)
I20261117 09:06:35.854302 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 6 FOLLOWER]: Advancing to term 7
I20261117 09:06:35.854941 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.855089 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 7 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:35.855293 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:35.855315 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 7 candidate_status { last_received { term: 6 index: 19 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:35.855403 22669 raft_consensus.cc:2415] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 97a11483cdc54c77975eedc94efd2768 in current term 7: Already voted for candidate a553d253650a4085a021354456c9d66a in this term.
I20261117 09:06:35.855930 22745 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 7.
I20261117 09:06:35.856112 22785 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters: a553d253650a4085a021354456c9d66a
I20261117 09:06:35.856199 22872 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 7 FOLLOWER]: Leader election won for term 7
I20261117 09:06:35.856245 22872 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 7 LEADER]: Becoming Leader. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:35.856293 22872 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 6.19, Last appended by leader: 19, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:35.856667 22669 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Refusing update from remote peer 97a11483cdc54c77975eedc94efd2768: Log matching property violated. Preceding OpId in replica: term: 6 index: 19. Preceding OpId from leader: term: 7 index: 20. (index mismatch)
I20261117 09:06:35.856739 22563 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 reported cstate change: term changed from 6 to 7, leader changed from 680033ff21ed4734971b949bf68b1895 (127.21.116.66) to 97a11483cdc54c77975eedc94efd2768 (127.21.116.67). New cstate: current_term: 7 leader_uuid: "97a11483cdc54c77975eedc94efd2768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:35.856833 22906 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20261117 09:06:35.856971 22745 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 7 FOLLOWER]: Refusing update from remote peer 97a11483cdc54c77975eedc94efd2768: Log matching property violated. Preceding OpId in replica: term: 6 index: 19. Preceding OpId from leader: term: 7 index: 20. (index mismatch)
I20261117 09:06:35.857151 22906 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20261117 09:06:36.132887 22903 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:36.132978 22903 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 7 FOLLOWER]: Advancing to term 8
I20261117 09:06:36.133620 22903 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.133764 22903 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 8 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:36.133972 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.133976 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:36.134182 22708 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 680033ff21ed4734971b949bf68b1895; no voters: 97a11483cdc54c77975eedc94efd2768, a553d253650a4085a021354456c9d66a
I20261117 09:06:36.134292 22903 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20261117 09:06:36.144520 22745 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Rejecting Update request from peer 97a11483cdc54c77975eedc94efd2768 for earlier term 7. Current term is 8. Ops: []
I20261117 09:06:36.144730 22907 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: INVALID_TERM, Last received: 7.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20261117 09:06:36.151893 22872 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 7 LEADER]: Stepping down as leader of term 7
I20261117 09:06:36.151947 22872 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:36.152004 22872 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 7.20, Last appended by leader: 20, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.152079 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 7 FOLLOWER]: Advancing to term 8
I20261117 09:06:36.157336 22874 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:36.157387 22874 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 7 FOLLOWER]: Advancing to term 8
I20261117 09:06:36.157887 22874 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.158004 22874 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 8 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:36.158136 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:36.158216 22745 raft_consensus.cc:2415] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a553d253650a4085a021354456c9d66a in current term 8: Already voted for candidate 680033ff21ed4734971b949bf68b1895 in this term.
I20261117 09:06:36.158138 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 8 candidate_status { last_received { term: 7 index: 20 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:36.158762 22820 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a553d253650a4085a021354456c9d66a in term 8.
I20261117 09:06:36.158905 22633 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768, a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895
I20261117 09:06:36.158983 22874 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 8 FOLLOWER]: Leader election won for term 8
I20261117 09:06:36.159072 22874 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 8 LEADER]: Becoming Leader. State: Replica: a553d253650a4085a021354456c9d66a, State: Running, Role: LEADER
I20261117 09:06:36.159111 22874 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 7.20, Last appended by leader: 20, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.159463 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 8 FOLLOWER]: Refusing update from remote peer a553d253650a4085a021354456c9d66a: Log matching property violated. Preceding OpId in replica: term: 7 index: 20. Preceding OpId from leader: term: 8 index: 21. (index mismatch)
I20261117 09:06:36.159468 22745 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Refusing update from remote peer a553d253650a4085a021354456c9d66a: Log matching property violated. Preceding OpId in replica: term: 7 index: 20. Preceding OpId from leader: term: 8 index: 21. (index mismatch)
I20261117 09:06:36.159548 22566 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a reported cstate change: term changed from 7 to 8, leader changed from 97a11483cdc54c77975eedc94efd2768 (127.21.116.67) to a553d253650a4085a021354456c9d66a (127.21.116.65). New cstate: current_term: 8 leader_uuid: "a553d253650a4085a021354456c9d66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:36.159682 22909 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20261117 09:06:36.159895 22910 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20261117 09:06:36.262228 22903 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a553d253650a4085a021354456c9d66a)
I20261117 09:06:36.262313 22903 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 8 FOLLOWER]: Advancing to term 9
I20261117 09:06:36.262955 22903 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.263099 22903 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 9 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:36.263286 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:36.263355 22820 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 8 FOLLOWER]: Advancing to term 9
I20261117 09:06:36.263293 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.263893 22820 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680033ff21ed4734971b949bf68b1895 in term 9.
I20261117 09:06:36.264068 22709 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters: a553d253650a4085a021354456c9d66a
I20261117 09:06:36.264153 22903 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 FOLLOWER]: Leader election won for term 9
I20261117 09:06:36.264248 22903 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 LEADER]: Becoming Leader. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:36.264290 22903 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 8.21, Last appended by leader: 21, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.264717 22669 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 8 LEADER]: Stepping down as leader of term 8
I20261117 09:06:36.264678 22563 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 reported cstate change: term changed from 8 to 9, leader changed from a553d253650a4085a021354456c9d66a (127.21.116.65) to 680033ff21ed4734971b949bf68b1895 (127.21.116.66). New cstate: current_term: 9 leader_uuid: "680033ff21ed4734971b949bf68b1895" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:36.264770 22669 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a553d253650a4085a021354456c9d66a, State: Running, Role: LEADER
I20261117 09:06:36.264815 22669 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 8.21, Last appended by leader: 21, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.264737 22745 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 LEADER]: Rejecting Update request from peer a553d253650a4085a021354456c9d66a for earlier term 8. Current term is 9. Ops: []
I20261117 09:06:36.264879 22669 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 8 FOLLOWER]: Advancing to term 9
I20261117 09:06:36.264974 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 9 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 8 index: 21. Preceding OpId from leader: term: 9 index: 22. (index mismatch)
I20261117 09:06:36.265506 22669 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 9 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 8 index: 21. Preceding OpId from leader: term: 9 index: 22. (index mismatch)
I20261117 09:06:36.265635 22911 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20261117 09:06:36.265733 22912 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20261117 09:06:36.477344 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 680033ff21ed4734971b949bf68b1895)
I20261117 09:06:36.477428 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 9 FOLLOWER]: Advancing to term 10
I20261117 09:06:36.479981 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.480127 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 10 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:36.480293 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.480306 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:36.480546 22785 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768; no voters: 680033ff21ed4734971b949bf68b1895, a553d253650a4085a021354456c9d66a
I20261117 09:06:36.480635 22872 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20261117 09:06:36.497903 22820 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Rejecting Update request from peer 680033ff21ed4734971b949bf68b1895 for earlier term 9. Current term is 10. Ops: []
I20261117 09:06:36.498193 22912 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: INVALID_TERM, Last received: 9.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20261117 09:06:36.532857 22911 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 LEADER]: Stepping down as leader of term 9
I20261117 09:06:36.532948 22911 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:36.533020 22911 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 9.22, Last appended by leader: 22, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.533131 22911 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 9 FOLLOWER]: Advancing to term 10
I20261117 09:06:36.577865 22909 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 9 FOLLOWER]: Starting leader election (detected failure of leader 680033ff21ed4734971b949bf68b1895)
I20261117 09:06:36.577950 22909 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 9 FOLLOWER]: Advancing to term 10
I20261117 09:06:36.578614 22909 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.578791 22909 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 10 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:36.578994 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:36.578994 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 10 candidate_status { last_received { term: 9 index: 22 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:36.579105 22820 raft_consensus.cc:2415] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a553d253650a4085a021354456c9d66a in current term 10: Already voted for candidate 97a11483cdc54c77975eedc94efd2768 in this term.
I20261117 09:06:36.579607 22745 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a553d253650a4085a021354456c9d66a in term 10.
I20261117 09:06:36.579802 22634 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, a553d253650a4085a021354456c9d66a; no voters: 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:36.579878 22909 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 10 FOLLOWER]: Leader election won for term 10
I20261117 09:06:36.579931 22909 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 10 LEADER]: Becoming Leader. State: Replica: a553d253650a4085a021354456c9d66a, State: Running, Role: LEADER
I20261117 09:06:36.579973 22909 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 9.22, Last appended by leader: 22, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.580379 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Refusing update from remote peer a553d253650a4085a021354456c9d66a: Log matching property violated. Preceding OpId in replica: term: 9 index: 22. Preceding OpId from leader: term: 10 index: 23. (index mismatch)
I20261117 09:06:36.580395 22745 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 10 FOLLOWER]: Refusing update from remote peer a553d253650a4085a021354456c9d66a: Log matching property violated. Preceding OpId in replica: term: 9 index: 22. Preceding OpId from leader: term: 10 index: 23. (index mismatch)
I20261117 09:06:36.580457 22565 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a reported cstate change: term changed from 9 to 10, leader changed from 680033ff21ed4734971b949bf68b1895 (127.21.116.66) to a553d253650a4085a021354456c9d66a (127.21.116.65). New cstate: current_term: 10 leader_uuid: "a553d253650a4085a021354456c9d66a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:36.580549 22910 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20261117 09:06:36.580762 22910 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20261117 09:06:36.681797 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a553d253650a4085a021354456c9d66a)
I20261117 09:06:36.681876 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 10 FOLLOWER]: Advancing to term 11
I20261117 09:06:36.682518 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.682653 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 11 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:36.682804 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:36.682840 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.683023 22783 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768; no voters: 680033ff21ed4734971b949bf68b1895, a553d253650a4085a021354456c9d66a
I20261117 09:06:36.683115 22872 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20261117 09:06:36.691748 22820 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 11 FOLLOWER]: Rejecting Update request from peer a553d253650a4085a021354456c9d66a for earlier term 10. Current term is 11. Ops: []
I20261117 09:06:36.691927 22910 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: INVALID_TERM, Last received: 10.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20261117 09:06:36.785496 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:36.785571 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 11 FOLLOWER]: Advancing to term 12
I20261117 09:06:36.786218 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.786367 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 12 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:36.786561 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:36.786561 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 12 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.786827 22785 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97a11483cdc54c77975eedc94efd2768; no voters: 680033ff21ed4734971b949bf68b1895, a553d253650a4085a021354456c9d66a
I20261117 09:06:36.786911 22872 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20261117 09:06:36.788770 22909 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 10 LEADER]: Stepping down as leader of term 10
I20261117 09:06:36.788821 22909 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a553d253650a4085a021354456c9d66a, State: Running, Role: LEADER
I20261117 09:06:36.788868 22909 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 10.23, Last appended by leader: 23, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.788945 22909 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 10 FOLLOWER]: Advancing to term 11
I20261117 09:06:36.794525 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.795704 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.797852 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.802008 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.810145 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.826325 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.856037 22911 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a553d253650a4085a021354456c9d66a)
I20261117 09:06:36.856096 22911 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 10 FOLLOWER]: Advancing to term 11
I20261117 09:06:36.856721 22911 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.856877 22911 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 11 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:36.857060 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:36.857061 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 11 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.857156 22820 raft_consensus.cc:2390] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 680033ff21ed4734971b949bf68b1895 for earlier term 11. Current term is 12.
I20261117 09:06:36.857369 22709 leader_election.cc:400] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 11 election: Vote denied by peer 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087) with higher term. Message: Invalid argument: T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 680033ff21ed4734971b949bf68b1895 for earlier term 11. Current term is 12.
I20261117 09:06:36.857409 22709 leader_election.cc:403] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20261117 09:06:36.857475 22911 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 11 FOLLOWER]: Advancing to term 12
I20261117 09:06:36.857744 22669 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680033ff21ed4734971b949bf68b1895 in term 11.
I20261117 09:06:36.858047 22911 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087) with higher term. Message: Invalid argument: T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 680033ff21ed4734971b949bf68b1895 for earlier term 11. Current term is 12.
I20261117 09:06:36.858503 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.903545 22872 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:36.903617 22872 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 12 FOLLOWER]: Advancing to term 13
I20261117 09:06:36.904150 22872 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.904287 22872 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 13 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), a553d253650a4085a021354456c9d66a (127.21.116.65:37587)
I20261117 09:06:36.904476 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 13 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:36.904543 22745 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 12 FOLLOWER]: Advancing to term 13
I20261117 09:06:36.904481 22669 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "97a11483cdc54c77975eedc94efd2768" candidate_term: 13 candidate_status { last_received { term: 10 index: 23 } } ignore_live_leader: false dest_uuid: "a553d253650a4085a021354456c9d66a"
I20261117 09:06:36.904577 22669 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 11 FOLLOWER]: Advancing to term 13
I20261117 09:06:36.905052 22669 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 13.
I20261117 09:06:36.905057 22745 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a11483cdc54c77975eedc94efd2768 in term 13.
I20261117 09:06:36.905237 22785 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters:
I20261117 09:06:36.905373 22872 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 13 FOLLOWER]: Leader election won for term 13
I20261117 09:06:36.905483 22872 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 13 LEADER]: Becoming Leader. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:36.905530 22872 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 10.23, Last appended by leader: 23, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:36.905915 22669 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 13 FOLLOWER]: Refusing update from remote peer 97a11483cdc54c77975eedc94efd2768: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 13 index: 24. (index mismatch)
I20261117 09:06:36.905937 22745 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 13 FOLLOWER]: Refusing update from remote peer 97a11483cdc54c77975eedc94efd2768: Log matching property violated. Preceding OpId in replica: term: 10 index: 23. Preceding OpId from leader: term: 13 index: 24. (index mismatch)
I20261117 09:06:36.905952 22566 catalog_manager.cc:5840] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 reported cstate change: term changed from 10 to 13, leader changed from a553d253650a4085a021354456c9d66a (127.21.116.65) to 97a11483cdc54c77975eedc94efd2768 (127.21.116.67). New cstate: current_term: 13 leader_uuid: "97a11483cdc54c77975eedc94efd2768" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:36.906100 22913 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20261117 09:06:36.906354 22914 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Connected to new peer: Peer: permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20261117 09:06:36.922659 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:36.922730 21969 log_reader.cc:385] T 1e3b46374d4047b8a68c84fda3badbc7: removed 1 log segments from log reader
I20261117 09:06:36.922755 21969 log.cc:1079] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Deleting log segment in path: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root/wals/1e3b46374d4047b8a68c84fda3badbc7/wal-000000001 (ops 1-15)
I20261117 09:06:36.923077 21969 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20261117 09:06:36.923123 21969 master.cc:585] Master@127.21.116.126:42017 shutting down...
I20261117 09:06:37.014412 22909 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 13 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:37.014494 22909 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 13 FOLLOWER]: Advancing to term 14
I20261117 09:06:37.015095 22909 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.015301 22909 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 14 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:37.015447 22745 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 14 candidate_status { last_received { term: 13 index: 24 } } ignore_live_leader: false dest_uuid: "680033ff21ed4734971b949bf68b1895"
I20261117 09:06:37.015447 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "a553d253650a4085a021354456c9d66a" candidate_term: 14 candidate_status { last_received { term: 13 index: 24 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
I20261117 09:06:37.015691 22634 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:37.015776 22909 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20261117 09:06:37.019258 22669 raft_consensus.cc:1250] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Rejecting Update request from peer 97a11483cdc54c77975eedc94efd2768 for earlier term 13. Current term is 14. Ops: []
I20261117 09:06:37.019433 22913 consensus_queue.cc:1060] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 }, Status: INVALID_TERM, Last received: 13.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20261117 09:06:37.019517 22913 raft_consensus.cc:3091] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 13 LEADER]: Stepping down as leader of term 13
I20261117 09:06:37.019546 22913 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 97a11483cdc54c77975eedc94efd2768, State: Running, Role: LEADER
I20261117 09:06:37.019603 22913 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 13.24, Last appended by leader: 24, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.019666 22913 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 13 FOLLOWER]: Advancing to term 14
I20261117 09:06:37.060614 21969 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 1 LEADER]: Raft consensus shutting down.
I20261117 09:06:37.060701 21969 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:37.060724 21969 tablet_replica.cc:333] T 00000000000000000000000000000000 P 70c99515af434a8f8995cc082ed2a9a5: stopping tablet replica
I20261117 09:06:37.072515 21969 master.cc:607] Master@127.21.116.126:42017 shutdown complete.
I20261117 09:06:37.073696 21969 tablet_server.cc:179] TabletServer@127.21.116.65:0 shutting down...
I20261117 09:06:37.098692 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:37.098856 21969 tablet_replica.cc:333] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: stopping tablet replica
I20261117 09:06:37.098919 21969 raft_consensus.cc:2261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:37.098968 21969 raft_consensus.cc:2290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:37.099720 22911 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 97a11483cdc54c77975eedc94efd2768)
I20261117 09:06:37.099776 22911 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 13 FOLLOWER]: Advancing to term 14
I20261117 09:06:37.100427 22911 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.100548 22911 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 14 election: Requested vote from peers a553d253650a4085a021354456c9d66a (127.21.116.65:37587), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
I20261117 09:06:37.100728 22820 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "1e3b46374d4047b8a68c84fda3badbc7" candidate_uuid: "680033ff21ed4734971b949bf68b1895" candidate_term: 14 candidate_status { last_received { term: 13 index: 24 } } ignore_live_leader: false dest_uuid: "97a11483cdc54c77975eedc94efd2768"
W20261117 09:06:37.101140 22708 leader_election.cc:336] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer a553d253650a4085a021354456c9d66a (127.21.116.65:37587): Network error: Client connection negotiation failed: client connection to 127.21.116.65:37587: connect: Connection refused (error 111)
I20261117 09:06:37.101311 21969 tablet_server.cc:196] TabletServer@127.21.116.65:0 shutdown complete.
I20261117 09:06:37.101526 22820 raft_consensus.cc:2490] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680033ff21ed4734971b949bf68b1895 in term 14.
I20261117 09:06:37.101753 22709 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768; no voters: a553d253650a4085a021354456c9d66a
I20261117 09:06:37.101840 22911 raft_consensus.cc:2826] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 14 FOLLOWER]: Leader election won for term 14
I20261117 09:06:37.101925 22911 raft_consensus.cc:705] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 14 LEADER]: Becoming Leader. State: Replica: 680033ff21ed4734971b949bf68b1895, State: Running, Role: LEADER
I20261117 09:06:37.101972 22911 consensus_queue.cc:238] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 13.24, Last appended by leader: 24, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.102234 21969 tablet_server.cc:179] TabletServer@127.21.116.66:0 shutting down...
I20261117 09:06:37.102456 22820 raft_consensus.cc:1285] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 14 FOLLOWER]: Refusing update from remote peer 680033ff21ed4734971b949bf68b1895: Log matching property violated. Preceding OpId in replica: term: 13 index: 24. Preceding OpId from leader: term: 14 index: 25. (index mismatch)
I20261117 09:06:37.102629 22919 consensus_queue.cc:1049] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20261117 09:06:37.102869 22708 consensus_peers.cc:597] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 -> Peer a553d253650a4085a021354456c9d66a (127.21.116.65:37587): Couldn't send request to peer a553d253650a4085a021354456c9d66a. Status: Network error: Client connection negotiation failed: client connection to 127.21.116.65:37587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261117 09:06:37.125535 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:37.125648 21969 tablet_replica.cc:333] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895: stopping tablet replica
I20261117 09:06:37.125701 21969 raft_consensus.cc:2261] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 14 LEADER]: Raft consensus shutting down.
I20261117 09:06:37.125759 21969 raft_consensus.cc:2290] T 1e3b46374d4047b8a68c84fda3badbc7 P 680033ff21ed4734971b949bf68b1895 [term 14 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:37.137398 21969 tablet_server.cc:196] TabletServer@127.21.116.66:0 shutdown complete.
I20261117 09:06:37.138178 21969 tablet_server.cc:179] TabletServer@127.21.116.67:0 shutting down...
I20261117 09:06:37.156827 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:37.156944 21969 tablet_replica.cc:333] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768: stopping tablet replica
I20261117 09:06:37.156991 21969 raft_consensus.cc:2261] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 14 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:37.157032 21969 raft_consensus.cc:2290] T 1e3b46374d4047b8a68c84fda3badbc7 P 97a11483cdc54c77975eedc94efd2768 [term 14 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:37.168571 21969 tablet_server.cc:196] TabletServer@127.21.116.67:0 shutdown complete.
I20261117 09:06:37.169239 21969 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20261117 09:06:37.169342 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
E20261117 09:06:37.169421 21969 multi_raft_batcher.cc:235] multi_raft_heartbeat_window_ms should not be more than raft_heartbeat_interval_ms / 2. , forcing multi_raft_heartbeat_window_ms = 50
W20261117 09:06:37.170356 22922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.170471 22925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.170535 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:37.170493 22923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.170715 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:37.170751 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:37.170768 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906397170767 us; error 0 us; skew 500 ppm
I20261117 09:06:37.171277 21969 webserver.cc:577] Webserver started at http://127.21.116.65:35687/ using document root <none> and password file <none>
I20261117 09:06:37.171356 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:37.171389 21969 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261117 09:06:37.171958 21969 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.173050 22932 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.173158 21969 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:37.173200 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "a553d253650a4085a021354456c9d66a"
format_stamp: "Formatted at 2026-11-17 09:06:34 on dist-test-slave-1b7w"
I20261117 09:06:37.173249 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5271
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:37.187764 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:37.187899 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:37.188000 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:37.188238 22939 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261117 09:06:37.188721 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261117 09:06:37.188757 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:37.188776 21969 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261117 09:06:37.189154 21969 ts_tablet_manager.cc:662] Registered 1 tablets
I20261117 09:06:37.189185 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.189194 22939 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Bootstrap starting.
I20261117 09:06:37.190459 22939 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261117 09:06:37.190656 22939 tablet_bootstrap.cc:492] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Bootstrap complete.
I20261117 09:06:37.190743 22939 ts_tablet_manager.cc:1449] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261117 09:06:37.190860 22939 raft_consensus.cc:360] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.190922 22939 raft_consensus.cc:748] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: a553d253650a4085a021354456c9d66a, State: Initialized, Role: FOLLOWER
I20261117 09:06:37.190989 22939 consensus_queue.cc:261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 13.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.191138 22939 ts_tablet_manager.cc:1480] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20261117 09:06:37.192077 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:37587
I20261117 09:06:37.193928 22998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:37587 every 8 connection(s)
W20261117 09:06:37.195573 22999 heartbeater.cc:413] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20261117 09:06:37.203864 21969 tablet_server.cc:179] TabletServer@127.21.116.65:37587 shutting down...
I20261117 09:06:37.268167 22978 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:37.268275 22978 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 14 FOLLOWER]: Advancing to term 15
I20261117 09:06:37.268936 22978 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.269127 22978 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 15 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
W20261117 09:06:37.269694 22934 leader_election.cc:336] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087): Network error: Client connection negotiation failed: client connection to 127.21.116.67:43087: connect: Connection refused (error 111)
W20261117 09:06:37.269775 22935 leader_election.cc:336] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487): Network error: Client connection negotiation failed: client connection to 127.21.116.66:39487: connect: Connection refused (error 111)
I20261117 09:06:37.269821 22935 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:37.269889 22978 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20261117 09:06:37.419373 22978 raft_consensus.cc:499] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:37.419469 22978 raft_consensus.cc:3096] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 15 FOLLOWER]: Advancing to term 16
I20261117 09:06:37.420115 22978 raft_consensus.cc:521] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680033ff21ed4734971b949bf68b1895" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 39487 } } peers { permanent_uuid: "a553d253650a4085a021354456c9d66a" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 37587 } } peers { permanent_uuid: "97a11483cdc54c77975eedc94efd2768" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 43087 } }
I20261117 09:06:37.420274 22978 leader_election.cc:290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 16 election: Requested vote from peers 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487), 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087)
W20261117 09:06:37.420791 22934 leader_election.cc:336] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 97a11483cdc54c77975eedc94efd2768 (127.21.116.67:43087): Network error: Client connection negotiation failed: client connection to 127.21.116.67:43087: connect: Connection refused (error 111)
W20261117 09:06:37.420856 22935 leader_election.cc:336] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 680033ff21ed4734971b949bf68b1895 (127.21.116.66:39487): Network error: Client connection negotiation failed: client connection to 127.21.116.66:39487: connect: Connection refused (error 111)
I20261117 09:06:37.420902 22935 leader_election.cc:304] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a553d253650a4085a021354456c9d66a; no voters: 680033ff21ed4734971b949bf68b1895, 97a11483cdc54c77975eedc94efd2768
I20261117 09:06:37.420966 22978 raft_consensus.cc:2771] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20261117 09:06:37.444346 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:37.444533 21969 tablet_replica.cc:333] stopping tablet replica
I20261117 09:06:37.444595 21969 raft_consensus.cc:2261] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 16 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:37.444641 21969 raft_consensus.cc:2290] T 1e3b46374d4047b8a68c84fda3badbc7 P a553d253650a4085a021354456c9d66a [term 16 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:37.456284 21969 tablet_server.cc:196] TabletServer@127.21.116.65:37587 shutdown complete.
[ OK ] TimestampAdvancementITest.TestNoOpAdvancesMvccSafeTimeOnBootstrap (3162 ms)
[ RUN ] TimestampAdvancementITest.Kudu2463Test
I20261117 09:06:37.460239 21969 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.116.126:37815
I20261117 09:06:37.460377 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:37.461495 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.461621 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.461695 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:37.461799 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.461895 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:37.461930 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:37.461949 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906397461950 us; error 0 us; skew 500 ppm
I20261117 09:06:37.462440 21969 webserver.cc:577] Webserver started at http://127.21.116.126:43365/ using document root <none> and password file <none>
I20261117 09:06:37.462517 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:37.462548 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:37.462656 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:37.462882 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/master-0-root/instance:
uuid: "ae5dc0d16d8e411498643be95f28c828"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.463606 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:37.463985 23014 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.464087 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.464125 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/master-0-root
uuid: "ae5dc0d16d8e411498643be95f28c828"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.464166 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:37.469447 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:37.469591 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:37.472322 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.126:37815
I20261117 09:06:37.472844 23066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.126:37815 every 8 connection(s)
I20261117 09:06:37.472949 23067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:37.473800 23067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828: Bootstrap starting.
I20261117 09:06:37.474050 23067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:37.474510 23067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828: No bootstrap required, opened a new log
I20261117 09:06:37.474617 23067 raft_consensus.cc:360] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5dc0d16d8e411498643be95f28c828" member_type: VOTER }
I20261117 09:06:37.474663 23067 raft_consensus.cc:386] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:37.474676 23067 raft_consensus.cc:748] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5dc0d16d8e411498643be95f28c828, State: Initialized, Role: FOLLOWER
I20261117 09:06:37.474720 23067 consensus_queue.cc:261] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5dc0d16d8e411498643be95f28c828" member_type: VOTER }
I20261117 09:06:37.474766 23067 raft_consensus.cc:400] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:37.474787 23067 raft_consensus.cc:499] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:37.474810 23067 raft_consensus.cc:3096] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:37.475250 23067 raft_consensus.cc:521] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5dc0d16d8e411498643be95f28c828" member_type: VOTER }
I20261117 09:06:37.475302 23067 leader_election.cc:304] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5dc0d16d8e411498643be95f28c828; no voters:
I20261117 09:06:37.475395 23067 leader_election.cc:290] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261117 09:06:37.475440 23070 raft_consensus.cc:2826] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:37.475528 23067 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [sys.catalog]: configured and running, proceeding with master startup.
I20261117 09:06:37.475558 23070 raft_consensus.cc:705] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 1 LEADER]: Becoming Leader. State: Replica: ae5dc0d16d8e411498643be95f28c828, State: Running, Role: LEADER
I20261117 09:06:37.475611 23070 consensus_queue.cc:238] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5dc0d16d8e411498643be95f28c828" member_type: VOTER }
I20261117 09:06:37.475791 23071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5dc0d16d8e411498643be95f28c828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5dc0d16d8e411498643be95f28c828" member_type: VOTER } }
I20261117 09:06:37.475886 23071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:37.475806 23072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5dc0d16d8e411498643be95f28c828. Latest consensus state: current_term: 1 leader_uuid: "ae5dc0d16d8e411498643be95f28c828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5dc0d16d8e411498643be95f28c828" member_type: VOTER } }
I20261117 09:06:37.476014 23072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:37.476094 23075 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261117 09:06:37.476219 23075 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261117 09:06:37.476745 21969 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261117 09:06:37.476755 23075 catalog_manager.cc:1398] Generated new cluster ID: 89d3272982e64075afe5e66f8e155524
I20261117 09:06:37.476817 23075 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261117 09:06:37.510706 23075 catalog_manager.cc:1421] Generated new certificate authority record
I20261117 09:06:37.511137 23075 catalog_manager.cc:1555] Loading token signing keys...
I20261117 09:06:37.515970 23075 catalog_manager.cc:6221] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828: Generated new TSK 0
I20261117 09:06:37.516073 23075 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261117 09:06:37.540627 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:37.541818 23088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.541875 23091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.541951 23089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.542299 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:37.542371 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:37.542428 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:37.542452 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906397542452 us; error 0 us; skew 500 ppm
I20261117 09:06:37.542970 21969 webserver.cc:577] Webserver started at http://127.21.116.65:40649/ using document root <none> and password file <none>
I20261117 09:06:37.543044 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:37.543090 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:37.543134 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:37.543395 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root/instance:
uuid: "1dde8fd442a34c9ab2ba416527328c1b"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.544183 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:37.544587 23096 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.544720 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.544762 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "1dde8fd442a34c9ab2ba416527328c1b"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.544804 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:37.553934 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:37.554112 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:37.554237 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:37.554442 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:37.554476 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.554499 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:37.554522 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.557013 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:39151
I20261117 09:06:37.557044 23159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:39151 every 8 connection(s)
I20261117 09:06:37.557327 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:37.558270 23164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.558310 23167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.558377 23165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.558537 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:37.558643 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:37.558676 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:37.558700 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906397558700 us; error 0 us; skew 500 ppm
I20261117 09:06:37.559222 21969 webserver.cc:577] Webserver started at http://127.21.116.66:42143/ using document root <none> and password file <none>
I20261117 09:06:37.559293 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:37.559335 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:37.559376 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:37.559597 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-1-root/instance:
uuid: "92b3a206716c455fb21c6fac969feeb7"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.560613 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20261117 09:06:37.560783 23160 heartbeater.cc:345] Connected to a master server at 127.21.116.126:37815
I20261117 09:06:37.560849 23160 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:37.560952 23160 heartbeater.cc:508] Master 127.21.116.126:37815 requested a full tablet report, sending...
I20261117 09:06:37.561074 23172 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.561177 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.561203 23031 ts_manager.cc:210] Registered new tserver with Master: 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65:39151)
I20261117 09:06:37.561223 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-1-root
uuid: "92b3a206716c455fb21c6fac969feeb7"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.561264 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:37.561744 23031 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44434
I20261117 09:06:37.575774 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:37.575971 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:37.576086 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:37.576251 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:37.576283 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.576313 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:37.576336 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.579056 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.66:42843
I20261117 09:06:37.579092 23235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.66:42843 every 8 connection(s)
I20261117 09:06:37.579391 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:37.580344 23239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.580404 23240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:37.580461 23242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:37.580648 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:37.580757 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:37.580787 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:37.580803 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906397580804 us; error 0 us; skew 500 ppm
I20261117 09:06:37.581283 21969 webserver.cc:577] Webserver started at http://127.21.116.67:37139/ using document root <none> and password file <none>
I20261117 09:06:37.581357 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:37.581393 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:37.581434 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:37.581667 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-2-root/instance:
uuid: "6da45b124e2f493d88fbead02cca8a53"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.582337 23236 heartbeater.cc:345] Connected to a master server at 127.21.116.126:37815
I20261117 09:06:37.582412 23236 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:37.582504 23236 heartbeater.cc:508] Master 127.21.116.126:37815 requested a full tablet report, sending...
I20261117 09:06:37.582607 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20261117 09:06:37.582710 23031 ts_manager.cc:210] Registered new tserver with Master: 92b3a206716c455fb21c6fac969feeb7 (127.21.116.66:42843)
I20261117 09:06:37.583037 23031 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44440
I20261117 09:06:37.583067 23247 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.583166 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.583209 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-2-root
uuid: "6da45b124e2f493d88fbead02cca8a53"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:37.583248 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:37.599166 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:37.599344 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:37.599471 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:37.599622 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:37.599647 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.599670 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:37.599704 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:37.602626 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.67:45533
I20261117 09:06:37.602669 23310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.67:45533 every 8 connection(s)
I20261117 09:06:37.605199 23311 heartbeater.cc:345] Connected to a master server at 127.21.116.126:37815
I20261117 09:06:37.605252 23311 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:37.605340 23311 heartbeater.cc:508] Master 127.21.116.126:37815 requested a full tablet report, sending...
I20261117 09:06:37.605530 23031 ts_manager.cc:210] Registered new tserver with Master: 6da45b124e2f493d88fbead02cca8a53 (127.21.116.67:45533)
I20261117 09:06:37.605863 23031 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44450
I20261117 09:06:37.606096 21969 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003308809s
I20261117 09:06:37.615824 23031 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44472:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261117 09:06:37.615945 23031 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261117 09:06:37.620091 23200 tablet_service.cc:1500] Processing CreateTablet for tablet cdd370b9c498403584c1af23a1bb8d90 (DEFAULT_TABLE table=test-workload [id=b89a287e44d5407395ab5c3b98c28e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:37.620213 23200 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd370b9c498403584c1af23a1bb8d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:37.620344 23275 tablet_service.cc:1500] Processing CreateTablet for tablet cdd370b9c498403584c1af23a1bb8d90 (DEFAULT_TABLE table=test-workload [id=b89a287e44d5407395ab5c3b98c28e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:37.620431 23275 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd370b9c498403584c1af23a1bb8d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:37.620774 23124 tablet_service.cc:1500] Processing CreateTablet for tablet cdd370b9c498403584c1af23a1bb8d90 (DEFAULT_TABLE table=test-workload [id=b89a287e44d5407395ab5c3b98c28e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:37.620877 23124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdd370b9c498403584c1af23a1bb8d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:37.621529 23332 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53: Bootstrap starting.
I20261117 09:06:37.621863 23333 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7: Bootstrap starting.
I20261117 09:06:37.621942 23332 tablet_bootstrap.cc:654] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:37.622208 23334 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Bootstrap starting.
I20261117 09:06:37.622340 23333 tablet_bootstrap.cc:654] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:37.622642 23334 tablet_bootstrap.cc:654] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:37.622870 23332 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53: No bootstrap required, opened a new log
I20261117 09:06:37.622923 23332 ts_tablet_manager.cc:1449] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:37.622932 23333 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7: No bootstrap required, opened a new log
I20261117 09:06:37.622977 23333 ts_tablet_manager.cc:1449] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:37.623061 23332 raft_consensus.cc:360] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.623127 23332 raft_consensus.cc:386] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:37.623111 23333 raft_consensus.cc:360] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.623142 23334 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: No bootstrap required, opened a new log
I20261117 09:06:37.623195 23334 ts_tablet_manager.cc:1449] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:37.623176 23333 raft_consensus.cc:386] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:37.623242 23333 raft_consensus.cc:748] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b3a206716c455fb21c6fac969feeb7, State: Initialized, Role: FOLLOWER
I20261117 09:06:37.623293 23333 consensus_queue.cc:261] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.623317 23334 raft_consensus.cc:360] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.623389 23334 raft_consensus.cc:386] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:37.623411 23334 raft_consensus.cc:748] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dde8fd442a34c9ab2ba416527328c1b, State: Initialized, Role: FOLLOWER
I20261117 09:06:37.623445 23333 ts_tablet_manager.cc:1480] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20261117 09:06:37.623150 23332 raft_consensus.cc:748] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da45b124e2f493d88fbead02cca8a53, State: Initialized, Role: FOLLOWER
I20261117 09:06:37.623468 23236 heartbeater.cc:500] Master 127.21.116.126:37815 was elected leader, sending a full tablet report...
I20261117 09:06:37.623517 23332 consensus_queue.cc:261] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.623672 23332 ts_tablet_manager.cc:1480] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:37.623749 23311 heartbeater.cc:500] Master 127.21.116.126:37815 was elected leader, sending a full tablet report...
I20261117 09:06:37.623453 23334 consensus_queue.cc:261] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.623915 23334 ts_tablet_manager.cc:1480] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:37.624022 23160 heartbeater.cc:500] Master 127.21.116.126:37815 was elected leader, sending a full tablet report...
I20261117 09:06:37.792995 23339 raft_consensus.cc:499] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261117 09:06:37.793104 23339 raft_consensus.cc:521] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.793275 23339 leader_election.cc:290] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b3a206716c455fb21c6fac969feeb7 (127.21.116.66:42843), 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65:39151)
I20261117 09:06:37.795885 23134 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "cdd370b9c498403584c1af23a1bb8d90" candidate_uuid: "6da45b124e2f493d88fbead02cca8a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dde8fd442a34c9ab2ba416527328c1b" is_pre_election: true
I20261117 09:06:37.795852 23210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "cdd370b9c498403584c1af23a1bb8d90" candidate_uuid: "6da45b124e2f493d88fbead02cca8a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b3a206716c455fb21c6fac969feeb7" is_pre_election: true
I20261117 09:06:37.795969 23134 raft_consensus.cc:2490] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da45b124e2f493d88fbead02cca8a53 in term 0.
I20261117 09:06:37.795971 23210 raft_consensus.cc:2490] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6da45b124e2f493d88fbead02cca8a53 in term 0.
I20261117 09:06:37.796121 23249 leader_election.cc:304] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dde8fd442a34c9ab2ba416527328c1b, 6da45b124e2f493d88fbead02cca8a53; no voters:
I20261117 09:06:37.796238 23339 raft_consensus.cc:2826] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261117 09:06:37.796284 23339 raft_consensus.cc:499] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261117 09:06:37.796309 23339 raft_consensus.cc:3096] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:37.796883 23339 raft_consensus.cc:521] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.796988 23339 leader_election.cc:290] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [CANDIDATE]: Term 1 election: Requested vote from peers 92b3a206716c455fb21c6fac969feeb7 (127.21.116.66:42843), 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65:39151)
I20261117 09:06:37.797160 23134 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "cdd370b9c498403584c1af23a1bb8d90" candidate_uuid: "6da45b124e2f493d88fbead02cca8a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dde8fd442a34c9ab2ba416527328c1b"
I20261117 09:06:37.797227 23134 raft_consensus.cc:3096] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:37.797818 23134 raft_consensus.cc:2490] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da45b124e2f493d88fbead02cca8a53 in term 1.
I20261117 09:06:37.797909 23210 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "cdd370b9c498403584c1af23a1bb8d90" candidate_uuid: "6da45b124e2f493d88fbead02cca8a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b3a206716c455fb21c6fac969feeb7"
I20261117 09:06:37.797950 23249 leader_election.cc:304] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dde8fd442a34c9ab2ba416527328c1b, 6da45b124e2f493d88fbead02cca8a53; no voters:
I20261117 09:06:37.797966 23210 raft_consensus.cc:3096] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:37.798018 23339 raft_consensus.cc:2826] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:37.798149 23339 raft_consensus.cc:705] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Becoming Leader. State: Replica: 6da45b124e2f493d88fbead02cca8a53, State: Running, Role: LEADER
I20261117 09:06:37.798202 23339 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.798728 23210 raft_consensus.cc:2490] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6da45b124e2f493d88fbead02cca8a53 in term 1.
I20261117 09:06:37.798765 23030 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6da45b124e2f493d88fbead02cca8a53 (127.21.116.67). New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:37.799031 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261117 09:06:37.799187 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261117 09:06:37.863143 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.863520 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20261117 09:06:37.863535 23209 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
I20261117 09:06:37.863699 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20261117 09:06:37.863785 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20261117 09:06:37.864259 23339 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index -1 to 18, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.864374 23209 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index -1 to 18, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.865303 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index -1 to 18, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.865301 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.18: config changed from index -1 to 18, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.865360 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.865669 23209 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
W20261117 09:06:37.865932 23031 catalog_manager.cc:5412] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 18: no extra replica candidate found for tablet cdd370b9c498403584c1af23a1bb8d90 (table test-workload [id=b89a287e44d5407395ab5c3b98c28e43]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20261117 09:06:37.866166 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch)
I20261117 09:06:37.866343 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20261117 09:06:37.866565 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20261117 09:06:37.866881 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.867053 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.867879 23029 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 18 to 19, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.868089 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.868436 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20261117 09:06:37.868600 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20261117 09:06:37.868602 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch)
I20261117 09:06:37.868623 23209 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index 18 to 19, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.869526 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20261117 09:06:37.869611 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.869719 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.869765 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.20: config changed from index 19 to 20, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.870518 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 19 to 20, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:37.870699 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.871048 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20261117 09:06:37.871236 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20261117 09:06:37.871306 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch)
I20261117 09:06:37.871486 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20261117 09:06:37.871742 23342 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.871857 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.871989 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index 20 to 21, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.872682 23029 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 20 to 21, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.872840 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.873198 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20261117 09:06:37.873283 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20261117 09:06:37.873381 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20261117 09:06:37.873453 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20261117 09:06:37.873968 23342 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.874082 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.874079 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.22: config changed from index 21 to 22, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.874758 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.874943 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b reported cstate change: config changed from index 21 to 22, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.875203 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20261117 09:06:37.875288 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20261117 09:06:37.875352 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20261117 09:06:37.875653 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20261117 09:06:37.875747 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.875865 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.876174 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.876613 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 22 to 23, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.877353 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.877656 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20261117 09:06:37.877671 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20261117 09:06:37.877826 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20261117 09:06:37.877913 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20261117 09:06:37.878283 23339 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.878432 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.879081 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.24: config changed from index 23 to 24, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.879315 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.879622 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20261117 09:06:37.879115 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 23 to 24, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.879797 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20261117 09:06:37.880180 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20261117 09:06:37.880314 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20261117 09:06:37.880306 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.880568 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.880982 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index 24 to 25, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.881206 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 24 to 25, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: UNKNOWN } } }
I20261117 09:06:37.881781 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.882135 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20261117 09:06:37.882143 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20261117 09:06:37.882302 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20261117 09:06:37.882377 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20261117 09:06:37.882783 23342 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.882928 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.883652 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 reported cstate change: config changed from index 25 to 26, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.883716 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.884053 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.884425 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20261117 09:06:37.884589 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20261117 09:06:37.884567 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20261117 09:06:37.884822 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20261117 09:06:37.885070 23339 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.885690 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.885771 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 26 to 27, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.885820 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.886822 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.887100 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20261117 09:06:37.887120 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20261117 09:06:37.887247 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20261117 09:06:37.887315 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20261117 09:06:37.887763 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.887883 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.887907 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.888451 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 27 to 28, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.888490 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.889005 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20261117 09:06:37.889005 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20261117 09:06:37.889202 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20261117 09:06:37.889346 23342 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20261117 09:06:37.889762 23339 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.889858 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.890618 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b reported cstate change: config changed from index 28 to 29, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.890632 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.891038 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.891331 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20261117 09:06:37.891394 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20261117 09:06:37.891532 23364 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20261117 09:06:37.891625 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20261117 09:06:37.892050 23342 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.892184 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.892737 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.892751 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 29 to 30, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.893028 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.893317 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20261117 09:06:37.893462 23364 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20261117 09:06:37.893510 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20261117 09:06:37.893718 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20261117 09:06:37.894042 23364 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.894134 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.894414 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.895035 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.895171 23016 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 20: aborting the task: latest config opid_index 30; task opid_index 20
I20261117 09:06:37.895321 23133 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20261117 09:06:37.895040 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 30 to 31, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.895522 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20261117 09:06:37.895701 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20261117 09:06:37.896076 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20261117 09:06:37.896538 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.896648 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.897266 23133 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.32: config changed from index 31 to 32, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.897312 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 31 to 32, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.897552 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.897900 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20261117 09:06:37.898012 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20261117 09:06:37.898092 23364 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20261117 09:06:37.898203 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20261117 09:06:37.898687 23339 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.898813 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.898842 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.899547 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b reported cstate change: config changed from index 32 to 33, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.899959 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.900318 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20261117 09:06:37.900337 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20261117 09:06:37.900489 23339 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20261117 09:06:37.900564 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20261117 09:06:37.901046 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.901151 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.901140 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.901737 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 33 to 34, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.901819 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.902215 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20261117 09:06:37.902372 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20261117 09:06:37.902446 23209 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 1 index: 35. (index mismatch)
I20261117 09:06:37.902729 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20261117 09:06:37.902877 23342 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.902993 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.903332 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.35: config changed from index 34 to 35, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.903734 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 reported cstate change: config changed from index 34 to 35, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:37.904098 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 1.35, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.904356 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20261117 09:06:37.904439 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 35. Preceding OpId from leader: term: 1 index: 36. (index mismatch)
I20261117 09:06:37.904564 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20261117 09:06:37.904683 23364 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20261117 09:06:37.905072 23344 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.905190 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.905879 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Committing config change with OpId 1.36: config changed from index 35 to 36, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.905915 23031 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 reported cstate change: config changed from index 35 to 36, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from VOTER to NON_VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: NON_VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.906582 23285 consensus_queue.cc:238] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:37.906966 23134 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 LEARNER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20261117 09:06:37.906966 23210 raft_consensus.cc:1285] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Refusing update from remote peer 6da45b124e2f493d88fbead02cca8a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20261117 09:06:37.907114 23344 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20261117 09:06:37.907186 23364 consensus_queue.cc:1049] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20261117 09:06:37.907713 23364 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.907796 23134 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.907814 23210 raft_consensus.cc:2983] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index 36 to 37, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.908378 21969 timestamp_advancement-itest.cc:267] GCing logs...
I20261117 09:06:37.908439 21969 log_reader.cc:385] T cdd370b9c498403584c1af23a1bb8d90: removed 1 log segments from log reader
I20261117 09:06:37.908459 21969 log.cc:1079] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Deleting log segment in path: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root/wals/cdd370b9c498403584c1af23a1bb8d90/wal-000000001 (ops 1-17)
I20261117 09:06:37.908576 23028 catalog_manager.cc:5840] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b reported cstate change: config changed from index 36 to 37, 1dde8fd442a34c9ab2ba416527328c1b (127.21.116.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6da45b124e2f493d88fbead02cca8a53" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } } }
I20261117 09:06:37.909152 21969 timestamp_advancement-itest.cc:274] Shutting down cluster...
I20261117 09:06:37.909192 21969 master.cc:585] Master@127.21.116.126:37815 shutting down...
I20261117 09:06:37.910985 23018 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 18: aborting the task: latest config opid_index 37; task opid_index 18
I20261117 09:06:37.915163 23017 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 28: aborting the task: latest config opid_index 37; task opid_index 28
I20261117 09:06:37.921321 23015 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 24: aborting the task: latest config opid_index 37; task opid_index 24
I20261117 09:06:37.935328 23017 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 26: aborting the task: latest config opid_index 37; task opid_index 26
I20261117 09:06:37.941355 23016 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 34: aborting the task: latest config opid_index 37; task opid_index 34
I20261117 09:06:37.943142 23018 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 30: aborting the task: latest config opid_index 37; task opid_index 30
I20261117 09:06:37.945479 23015 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 32: aborting the task: latest config opid_index 37; task opid_index 32
I20261117 09:06:37.961642 23015 catalog_manager.cc:5281] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cdd370b9c498403584c1af23a1bb8d90 with cas_config_opid_index 36: aborting the task: latest config opid_index 37; task opid_index 36
I20261117 09:06:37.970345 21969 raft_consensus.cc:2261] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 1 LEADER]: Raft consensus shutting down.
I20261117 09:06:37.970428 21969 raft_consensus.cc:2290] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:37.970502 21969 tablet_replica.cc:333] T 00000000000000000000000000000000 P ae5dc0d16d8e411498643be95f28c828: stopping tablet replica
I20261117 09:06:37.982645 21969 master.cc:607] Master@127.21.116.126:37815 shutdown complete.
I20261117 09:06:37.983882 21969 tablet_server.cc:179] TabletServer@127.21.116.65:0 shutting down...
I20261117 09:06:38.057713 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.057911 21969 tablet_replica.cc:333] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: stopping tablet replica
I20261117 09:06:38.057986 21969 raft_consensus.cc:2261] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:38.058068 21969 raft_consensus.cc:2290] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.069769 21969 tablet_server.cc:196] TabletServer@127.21.116.65:0 shutdown complete.
I20261117 09:06:38.070766 21969 tablet_server.cc:179] TabletServer@127.21.116.66:0 shutting down...
I20261117 09:06:38.079682 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.079845 21969 tablet_replica.cc:333] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7: stopping tablet replica
I20261117 09:06:38.079908 21969 raft_consensus.cc:2261] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:38.080000 21969 raft_consensus.cc:2290] T cdd370b9c498403584c1af23a1bb8d90 P 92b3a206716c455fb21c6fac969feeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.091609 21969 tablet_server.cc:196] TabletServer@127.21.116.66:0 shutdown complete.
I20261117 09:06:38.092326 21969 tablet_server.cc:179] TabletServer@127.21.116.67:0 shutting down...
I20261117 09:06:38.103221 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.103379 21969 tablet_replica.cc:333] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53: stopping tablet replica
I20261117 09:06:38.103425 21969 raft_consensus.cc:2261] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 LEADER]: Raft consensus shutting down.
I20261117 09:06:38.103489 21969 raft_consensus.cc:2290] T cdd370b9c498403584c1af23a1bb8d90 P 6da45b124e2f493d88fbead02cca8a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.115468 21969 tablet_server.cc:196] TabletServer@127.21.116.67:0 shutdown complete.
I20261117 09:06:38.116143 21969 timestamp_advancement-itest.cc:282] Restarting single tablet server...
I20261117 09:06:38.116252 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.117332 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.117343 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.117445 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.117587 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:38.117676 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.117705 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.117722 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398117721 us; error 0 us; skew 500 ppm
I20261117 09:06:38.118216 21969 webserver.cc:577] Webserver started at http://127.21.116.65:40649/ using document root <none> and password file <none>
I20261117 09:06:38.118292 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.118322 21969 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261117 09:06:38.118904 21969 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.119982 23376 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.120081 21969 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:38.120132 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "1dde8fd442a34c9ab2ba416527328c1b"
format_stamp: "Formatted at 2026-11-17 09:06:37 on dist-test-slave-1b7w"
I20261117 09:06:38.120183 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.Kudu2463Test.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 5
Total live bytes: 5316
Total live bytes (after alignment): 24576
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.131182 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.131330 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.131448 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:38.132097 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261117 09:06:38.132131 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:38.132485 21969 ts_tablet_manager.cc:662] Registered 1 tablets
I20261117 09:06:38.132511 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.132519 23383 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Bootstrap starting.
I20261117 09:06:38.134099 23383 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261117 09:06:38.134294 23383 tablet_bootstrap.cc:492] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Bootstrap complete.
I20261117 09:06:38.134404 23383 ts_tablet_manager.cc:1449] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261117 09:06:38.134511 23383 raft_consensus.cc:360] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:38.134570 23383 raft_consensus.cc:748] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dde8fd442a34c9ab2ba416527328c1b, State: Initialized, Role: FOLLOWER
I20261117 09:06:38.134617 23383 consensus_queue.cc:261] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 37, Last appended: 1.37, Last appended by leader: 37, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "6da45b124e2f493d88fbead02cca8a53" member_type: VOTER last_known_addr { host: "127.21.116.67" port: 45533 } } peers { permanent_uuid: "92b3a206716c455fb21c6fac969feeb7" member_type: VOTER last_known_addr { host: "127.21.116.66" port: 42843 } } peers { permanent_uuid: "1dde8fd442a34c9ab2ba416527328c1b" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 39151 } }
I20261117 09:06:38.134764 23383 ts_tablet_manager.cc:1480] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.135406 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:39151
I20261117 09:06:38.136816 23442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:39151 every 8 connection(s)
W20261117 09:06:38.137611 23443 heartbeater.cc:413] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20261117 09:06:38.141407 23396 tablet_service.cc:2949] Rejecting scan request for tablet cdd370b9c498403584c1af23a1bb8d90: Uninitialized: clean time has not yet been initialized
I20261117 09:06:38.141563 21969 timestamp_advancement-itest.cc:447] Got response: error { code: TABLET_NOT_RUNNING status { code: UNINITIALIZED message: "clean time has not yet been initialized" } }
I20261117 09:06:38.141793 21969 tablet_server.cc:179] TabletServer@127.21.116.65:39151 shutting down...
I20261117 09:06:38.387279 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.387494 21969 tablet_replica.cc:333] stopping tablet replica
I20261117 09:06:38.387565 21969 raft_consensus.cc:2261] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20261117 09:06:38.387619 21969 raft_consensus.cc:2290] T cdd370b9c498403584c1af23a1bb8d90 P 1dde8fd442a34c9ab2ba416527328c1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.399094 21969 tablet_server.cc:196] TabletServer@127.21.116.65:39151 shutdown complete.
[ OK ] TimestampAdvancementITest.Kudu2463Test (942 ms)
[ RUN ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp
I20261117 09:06:38.402925 21969 timestamp_advancement-itest.cc:460] Running with 1 replicas
I20261117 09:06:38.403061 21969 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.116.126:39259
I20261117 09:06:38.403205 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.404222 23450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.404353 23448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.404379 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.404430 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:38.404551 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.404584 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.404600 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398404601 us; error 0 us; skew 500 ppm
I20261117 09:06:38.405076 21969 webserver.cc:577] Webserver started at http://127.21.116.126:33059/ using document root <none> and password file <none>
I20261117 09:06:38.405144 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.405185 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:38.405225 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:38.405455 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root/instance:
uuid: "0861155fb11744509443d123e5e28a41"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.406230 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:38.406760 23455 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.406883 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.406924 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root
uuid: "0861155fb11744509443d123e5e28a41"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.406967 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.416370 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.416512 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.418802 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.126:39259
I20261117 09:06:38.419267 23507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.126:39259 every 8 connection(s)
I20261117 09:06:38.419391 23508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:38.420261 23508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Bootstrap starting.
I20261117 09:06:38.420502 23508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:38.420912 23508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: No bootstrap required, opened a new log
I20261117 09:06:38.421015 23508 raft_consensus.cc:360] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.421062 23508 raft_consensus.cc:386] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:38.421080 23508 raft_consensus.cc:748] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0861155fb11744509443d123e5e28a41, State: Initialized, Role: FOLLOWER
I20261117 09:06:38.421128 23508 consensus_queue.cc:261] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.421170 23508 raft_consensus.cc:400] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:38.421195 23508 raft_consensus.cc:499] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:38.421218 23508 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:38.421650 23508 raft_consensus.cc:521] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.421702 23508 leader_election.cc:304] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0861155fb11744509443d123e5e28a41; no voters:
I20261117 09:06:38.421793 23508 leader_election.cc:290] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261117 09:06:38.421830 23511 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:38.421916 23508 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: configured and running, proceeding with master startup.
I20261117 09:06:38.421952 23511 raft_consensus.cc:705] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 LEADER]: Becoming Leader. State: Replica: 0861155fb11744509443d123e5e28a41, State: Running, Role: LEADER
I20261117 09:06:38.422039 23511 consensus_queue.cc:238] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.422235 23512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0861155fb11744509443d123e5e28a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER } }
I20261117 09:06:38.422335 23512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:38.422235 23513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0861155fb11744509443d123e5e28a41. Latest consensus state: current_term: 1 leader_uuid: "0861155fb11744509443d123e5e28a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER } }
I20261117 09:06:38.422519 23513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:38.423070 21969 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20261117 09:06:38.423158 23527 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261117 09:06:38.423194 23527 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261117 09:06:38.423245 23518 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261117 09:06:38.423328 23518 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261117 09:06:38.423815 23518 catalog_manager.cc:1398] Generated new cluster ID: fafff15bf6e143a782d6b9de1d1e0e53
I20261117 09:06:38.423866 23518 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261117 09:06:38.426522 23518 catalog_manager.cc:1421] Generated new certificate authority record
I20261117 09:06:38.426877 23518 catalog_manager.cc:1555] Loading token signing keys...
I20261117 09:06:38.442521 23518 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Generated new TSK 0
I20261117 09:06:38.442600 23518 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261117 09:06:38.454816 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.455982 23529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.456075 23530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.456100 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:38.456203 23532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.456363 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.456393 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.456409 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398456409 us; error 0 us; skew 500 ppm
I20261117 09:06:38.456933 21969 webserver.cc:577] Webserver started at http://127.21.116.65:37607/ using document root <none> and password file <none>
I20261117 09:06:38.457005 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.457047 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:38.457093 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:38.457336 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root/instance:
uuid: "5692d6cbd46246fe83bad08563c4c8d1"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.458102 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:38.458554 23537 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.458666 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.458704 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "5692d6cbd46246fe83bad08563c4c8d1"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.458746 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.483111 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.483274 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.483394 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:38.483558 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:38.483583 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.483608 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:38.483628 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.486122 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:45603
I20261117 09:06:38.486171 23600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:45603 every 8 connection(s)
I20261117 09:06:38.489104 23601 heartbeater.cc:345] Connected to a master server at 127.21.116.126:39259
I20261117 09:06:38.489171 23601 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:38.489245 23601 heartbeater.cc:508] Master 127.21.116.126:39259 requested a full tablet report, sending...
I20261117 09:06:38.489464 23472 ts_manager.cc:210] Registered new tserver with Master: 5692d6cbd46246fe83bad08563c4c8d1 (127.21.116.65:45603)
I20261117 09:06:38.489602 21969 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003314581s
I20261117 09:06:38.490067 23472 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54574
I20261117 09:06:38.499848 23472 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54590:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261117 09:06:38.503269 23565 tablet_service.cc:1500] Processing CreateTablet for tablet 7f64b7f1053e4b7c84d4a4ceb1dd3d67 (DEFAULT_TABLE table=test-workload [id=12b757b219274ca4b3e505527a6fba50]), partition=RANGE (key) PARTITION UNBOUNDED
I20261117 09:06:38.503374 23565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f64b7f1053e4b7c84d4a4ceb1dd3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20261117 09:06:38.504513 23619 tablet_bootstrap.cc:492] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Bootstrap starting.
I20261117 09:06:38.504807 23619 tablet_bootstrap.cc:654] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Neither blocks nor log segments found. Creating new log.
I20261117 09:06:38.505369 23619 tablet_bootstrap.cc:492] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: No bootstrap required, opened a new log
I20261117 09:06:38.505414 23619 ts_tablet_manager.cc:1449] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:38.505504 23619 raft_consensus.cc:360] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.505550 23619 raft_consensus.cc:386] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261117 09:06:38.505563 23619 raft_consensus.cc:748] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5692d6cbd46246fe83bad08563c4c8d1, State: Initialized, Role: FOLLOWER
I20261117 09:06:38.505617 23619 consensus_queue.cc:261] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.505657 23619 raft_consensus.cc:400] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:38.505676 23619 raft_consensus.cc:499] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:38.505700 23619 raft_consensus.cc:3096] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 0 FOLLOWER]: Advancing to term 1
I20261117 09:06:38.506254 23619 raft_consensus.cc:521] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.506317 23619 leader_election.cc:304] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5692d6cbd46246fe83bad08563c4c8d1; no voters:
I20261117 09:06:38.506438 23619 leader_election.cc:290] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261117 09:06:38.506490 23621 raft_consensus.cc:2826] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Leader election won for term 1
I20261117 09:06:38.506562 23619 ts_tablet_manager.cc:1480] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:38.506611 23621 raft_consensus.cc:705] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 LEADER]: Becoming Leader. State: Replica: 5692d6cbd46246fe83bad08563c4c8d1, State: Running, Role: LEADER
I20261117 09:06:38.506657 23601 heartbeater.cc:500] Master 127.21.116.126:39259 was elected leader, sending a full tablet report...
I20261117 09:06:38.506682 23621 consensus_queue.cc:238] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.507068 23472 catalog_manager.cc:5840] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5692d6cbd46246fe83bad08563c4c8d1 (127.21.116.65). New cstate: current_term: 1 leader_uuid: "5692d6cbd46246fe83bad08563c4c8d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:38.509460 21969 tablet_server.cc:179] TabletServer@127.21.116.65:0 shutting down...
I20261117 09:06:38.511451 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.511583 21969 tablet_replica.cc:333] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: stopping tablet replica
I20261117 09:06:38.511637 21969 raft_consensus.cc:2261] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 LEADER]: Raft consensus shutting down.
I20261117 09:06:38.511677 21969 raft_consensus.cc:2290] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.522940 21969 tablet_server.cc:196] TabletServer@127.21.116.65:0 shutdown complete.
I20261117 09:06:38.523729 21969 master.cc:585] Master@127.21.116.126:39259 shutting down...
I20261117 09:06:38.525651 21969 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 LEADER]: Raft consensus shutting down.
I20261117 09:06:38.525709 21969 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.525770 21969 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: stopping tablet replica
I20261117 09:06:38.537351 21969 master.cc:607] Master@127.21.116.126:39259 shutdown complete.
I20261117 09:06:38.538846 21969 timestamp_advancement-itest.cc:460] Running with 3 replicas
I20261117 09:06:38.539004 21969 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.116.126:42659
I20261117 09:06:38.539113 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.540300 23629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.540400 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:38.540414 23626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.540429 23627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.540601 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.540634 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.540650 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398540650 us; error 0 us; skew 500 ppm
I20261117 09:06:38.541116 21969 webserver.cc:577] Webserver started at http://127.21.116.126:39765/ using document root <none> and password file <none>
I20261117 09:06:38.541185 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.541217 21969 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261117 09:06:38.541688 21969 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.542081 23634 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.542191 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.542240 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root
uuid: "0861155fb11744509443d123e5e28a41"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.542279 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.560709 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.560853 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.563019 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.126:42659
I20261117 09:06:38.563802 23686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.126:42659 every 8 connection(s)
I20261117 09:06:38.564240 23687 sys_catalog.cc:263] Verifying existing consensus state
I20261117 09:06:38.564549 23687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Bootstrap starting.
I20261117 09:06:38.565771 23687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261117 09:06:38.565948 23687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Bootstrap complete.
I20261117 09:06:38.566102 23687 raft_consensus.cc:360] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.566150 23687 raft_consensus.cc:748] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0861155fb11744509443d123e5e28a41, State: Initialized, Role: FOLLOWER
I20261117 09:06:38.566183 23687 consensus_queue.cc:261] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.566219 23687 raft_consensus.cc:400] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:38.566238 23687 raft_consensus.cc:499] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:38.566282 23687 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:38.566772 23687 raft_consensus.cc:521] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.566835 23687 leader_election.cc:304] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0861155fb11744509443d123e5e28a41; no voters:
I20261117 09:06:38.566937 23687 leader_election.cc:290] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [CANDIDATE]: Term 2 election: Requested vote from peers
I20261117 09:06:38.566982 23690 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 2 FOLLOWER]: Leader election won for term 2
I20261117 09:06:38.567058 23687 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: configured and running, proceeding with master startup.
I20261117 09:06:38.567116 23690 raft_consensus.cc:705] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 2 LEADER]: Becoming Leader. State: Replica: 0861155fb11744509443d123e5e28a41, State: Running, Role: LEADER
I20261117 09:06:38.567169 23690 consensus_queue.cc:238] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER }
I20261117 09:06:38.567387 23691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0861155fb11744509443d123e5e28a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER } }
I20261117 09:06:38.567389 23692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0861155fb11744509443d123e5e28a41. Latest consensus state: current_term: 2 leader_uuid: "0861155fb11744509443d123e5e28a41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0861155fb11744509443d123e5e28a41" member_type: VOTER } }
I20261117 09:06:38.567507 23692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:38.567461 23691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [sys.catalog]: This master's current role is: LEADER
I20261117 09:06:38.567775 23697 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261117 09:06:38.567963 23697 catalog_manager.cc:694] Loaded metadata for table test-workload [id=12b757b219274ca4b3e505527a6fba50]
I20261117 09:06:38.568069 23697 tablet_loader.cc:96] loaded metadata for tablet 7f64b7f1053e4b7c84d4a4ceb1dd3d67 (table test-workload [id=12b757b219274ca4b3e505527a6fba50])
I20261117 09:06:38.568109 23697 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261117 09:06:38.568174 23697 catalog_manager.cc:1310] Loaded cluster ID: fafff15bf6e143a782d6b9de1d1e0e53
I20261117 09:06:38.568199 21969 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20261117 09:06:38.568200 23697 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261117 09:06:38.568975 23697 catalog_manager.cc:1555] Loading token signing keys...
I20261117 09:06:38.569077 23697 catalog_manager.cc:6232] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41: Loaded TSK: 0
I20261117 09:06:38.569185 23697 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261117 09:06:38.569422 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.570410 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.570492 23707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.570492 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.570600 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:38.570673 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.570701 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.570724 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398570723 us; error 0 us; skew 500 ppm
I20261117 09:06:38.571136 21969 webserver.cc:577] Webserver started at http://127.21.116.65:44969/ using document root <none> and password file <none>
I20261117 09:06:38.571203 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.571235 21969 fs_manager.cc:365] Using existing metadata directory in first data directory
I20261117 09:06:38.571664 21969 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.572011 23715 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.572108 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.572144 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root
uuid: "5692d6cbd46246fe83bad08563c4c8d1"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.572181 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.581391 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.581524 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.581635 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:38.581849 23722 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261117 09:06:38.582317 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261117 09:06:38.582350 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:38.582374 21969 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261117 09:06:38.582759 21969 ts_tablet_manager.cc:662] Registered 1 tablets
I20261117 09:06:38.582787 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.582791 23722 tablet_bootstrap.cc:492] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Bootstrap starting.
I20261117 09:06:38.583722 23722 tablet_bootstrap.cc:492] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261117 09:06:38.583899 23722 tablet_bootstrap.cc:492] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Bootstrap complete.
I20261117 09:06:38.583979 23722 ts_tablet_manager.cc:1449] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:38.584079 23722 raft_consensus.cc:360] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.584125 23722 raft_consensus.cc:748] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5692d6cbd46246fe83bad08563c4c8d1, State: Initialized, Role: FOLLOWER
I20261117 09:06:38.584184 23722 consensus_queue.cc:261] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.584232 23722 raft_consensus.cc:400] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261117 09:06:38.584259 23722 raft_consensus.cc:499] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261117 09:06:38.584276 23722 raft_consensus.cc:3096] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 1 FOLLOWER]: Advancing to term 2
I20261117 09:06:38.584786 23722 raft_consensus.cc:521] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.584844 23722 leader_election.cc:304] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5692d6cbd46246fe83bad08563c4c8d1; no voters:
I20261117 09:06:38.584936 23722 leader_election.cc:290] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20261117 09:06:38.584995 23765 raft_consensus.cc:2826] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 2 FOLLOWER]: Leader election won for term 2
I20261117 09:06:38.585060 23722 ts_tablet_manager.cc:1480] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20261117 09:06:38.585073 23765 raft_consensus.cc:705] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 2 LEADER]: Becoming Leader. State: Replica: 5692d6cbd46246fe83bad08563c4c8d1, State: Running, Role: LEADER
I20261117 09:06:38.585114 23765 consensus_queue.cc:238] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } }
I20261117 09:06:38.585918 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.65:42023
I20261117 09:06:38.585947 23784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.65:42023 every 8 connection(s)
I20261117 09:06:38.586234 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.587108 23789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.587239 23790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.587306 21969 server_base.cc:1061] running on GCE node
W20261117 09:06:38.587360 23792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.587494 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.587531 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.587560 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398587561 us; error 0 us; skew 500 ppm
I20261117 09:06:38.588084 21969 webserver.cc:577] Webserver started at http://127.21.116.66:40633/ using document root <none> and password file <none>
I20261117 09:06:38.588157 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.588199 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:38.588248 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:38.588471 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-1-root/instance:
uuid: "51c3f8181e8947fbae69b9d6a3ad1e91"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.589430 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261117 09:06:38.589711 23785 heartbeater.cc:345] Connected to a master server at 127.21.116.126:42659
I20261117 09:06:38.589772 23785 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:38.589860 23785 heartbeater.cc:508] Master 127.21.116.126:42659 requested a full tablet report, sending...
I20261117 09:06:38.589939 23797 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.590095 23651 ts_manager.cc:210] Registered new tserver with Master: 5692d6cbd46246fe83bad08563c4c8d1 (127.21.116.65:42023)
I20261117 09:06:38.590144 23651 catalog_manager.cc:5840] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "5692d6cbd46246fe83bad08563c4c8d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5692d6cbd46246fe83bad08563c4c8d1" member_type: VOTER last_known_addr { host: "127.21.116.65" port: 45603 } health_report { overall_health: HEALTHY } } }
I20261117 09:06:38.590323 21969 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20261117 09:06:38.590355 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-1-root
uuid: "51c3f8181e8947fbae69b9d6a3ad1e91"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.590410 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.591415 23651 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44314
I20261117 09:06:38.591794 23785 heartbeater.cc:500] Master 127.21.116.126:42659 was elected leader, sending a full tablet report...
I20261117 09:06:38.596302 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.596417 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.596529 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:38.596653 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:38.596675 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.596696 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:38.596711 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.599582 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.66:37623
I20261117 09:06:38.599638 23861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.66:37623 every 8 connection(s)
I20261117 09:06:38.599866 21969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261117 09:06:38.600821 23865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.600947 23866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261117 09:06:38.601227 23868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261117 09:06:38.601446 21969 server_base.cc:1061] running on GCE node
I20261117 09:06:38.601565 21969 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20261117 09:06:38.601591 21969 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20261117 09:06:38.601605 21969 hybrid_clock.cc:648] HybridClock initialized: now 1794906398601605 us; error 0 us; skew 500 ppm
I20261117 09:06:38.602254 21969 webserver.cc:577] Webserver started at http://127.21.116.67:46757/ using document root <none> and password file <none>
I20261117 09:06:38.602334 21969 fs_manager.cc:362] Metadata directory not provided
I20261117 09:06:38.602373 21969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261117 09:06:38.602428 21969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261117 09:06:38.602607 23862 heartbeater.cc:345] Connected to a master server at 127.21.116.126:42659
I20261117 09:06:38.602659 23862 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:38.602677 21969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-2-root/instance:
uuid: "d7d34c30b85243a187bab967246c25fb"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.602756 23862 heartbeater.cc:508] Master 127.21.116.126:42659 requested a full tablet report, sending...
I20261117 09:06:38.602945 23651 ts_manager.cc:210] Registered new tserver with Master: 51c3f8181e8947fbae69b9d6a3ad1e91 (127.21.116.66:37623)
I20261117 09:06:38.603274 23651 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44320
I20261117 09:06:38.603569 21969 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20261117 09:06:38.603991 23873 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.604087 21969 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.604131 21969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-2-root
uuid: "d7d34c30b85243a187bab967246c25fb"
format_stamp: "Formatted at 2026-11-17 09:06:38 on dist-test-slave-1b7w"
I20261117 09:06:38.604169 21969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskHlySBD/test-tmp/timestamp_advancement-itest.0.TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp.1794906391332839-21969-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261117 09:06:38.609701 21969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261117 09:06:38.609894 21969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261117 09:06:38.610013 21969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261117 09:06:38.610177 21969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261117 09:06:38.610201 21969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.610266 21969 ts_tablet_manager.cc:662] Registered 0 tablets
I20261117 09:06:38.610291 21969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261117 09:06:38.613195 21969 rpc_server.cc:307] RPC server started. Bound to: 127.21.116.67:46801
I20261117 09:06:38.613246 23936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.116.67:46801 every 8 connection(s)
I20261117 09:06:38.615648 23937 heartbeater.cc:345] Connected to a master server at 127.21.116.126:42659
I20261117 09:06:38.615694 23937 heartbeater.cc:462] Registering TS with master...
I20261117 09:06:38.615769 23937 heartbeater.cc:508] Master 127.21.116.126:42659 requested a full tablet report, sending...
I20261117 09:06:38.615960 23651 ts_manager.cc:210] Registered new tserver with Master: d7d34c30b85243a187bab967246c25fb (127.21.116.67:46801)
I20261117 09:06:38.616276 23651 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44322
I20261117 09:06:38.616600 21969 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003248836s
I20261117 09:06:38.626719 21969 test_workload.cc:405] TestWorkload: Skipping table creation because table test-workload already exists
I20261117 09:06:38.626778 21969 tablet_server.cc:179] TabletServer@127.21.116.65:0 shutting down...
I20261117 09:06:38.628662 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.628845 21969 tablet_replica.cc:333] stopping tablet replica
I20261117 09:06:38.628906 21969 raft_consensus.cc:2261] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 2 LEADER]: Raft consensus shutting down.
I20261117 09:06:38.628949 21969 raft_consensus.cc:2290] T 7f64b7f1053e4b7c84d4a4ceb1dd3d67 P 5692d6cbd46246fe83bad08563c4c8d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.640336 21969 tablet_server.cc:196] TabletServer@127.21.116.65:0 shutdown complete.
I20261117 09:06:38.641043 21969 tablet_server.cc:179] TabletServer@127.21.116.66:0 shutting down...
I20261117 09:06:38.643607 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.654707 21969 tablet_server.cc:196] TabletServer@127.21.116.66:0 shutdown complete.
I20261117 09:06:38.655488 21969 tablet_server.cc:179] TabletServer@127.21.116.67:0 shutting down...
I20261117 09:06:38.657727 21969 ts_tablet_manager.cc:1553] Shutting down tablet manager...
I20261117 09:06:38.668785 21969 tablet_server.cc:196] TabletServer@127.21.116.67:0 shutdown complete.
I20261117 09:06:38.669396 21969 master.cc:585] Master@127.21.116.126:42659 shutting down...
I20261117 09:06:38.671438 21969 raft_consensus.cc:2261] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 2 LEADER]: Raft consensus shutting down.
I20261117 09:06:38.671495 21969 raft_consensus.cc:2290] T 00000000000000000000000000000000 P 0861155fb11744509443d123e5e28a41 [term 2 FOLLOWER]: Raft consensus is shut down!
I20261117 09:06:38.671562 21969 tablet_replica.cc:333] stopping tablet replica
I20261117 09:06:38.683076 21969 master.cc:607] Master@127.21.116.126:42659 shutdown complete.
[ OK ] TimestampAdvancementITest.TestTimestampsAdvancedFromRaftNoOp (283 ms)
[----------] 4 tests from TimestampAdvancementITest (7349 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (7349 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TimestampAdvancementITest.TestUpgradeFromOlderCorruptedData
1 FAILED TEST
I20261117 09:06:38.686761 21969 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5412: suppressed but not reported on 8 messages since previous log ~0 seconds ago
I20261117 09:06:38.686794 21969 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:648: suppressed but not reported on 19 messages since previous log ~4 seconds ago
I20261117 09:06:38.686811 21969 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 15 messages since previous log ~4 seconds ago
I20261117 09:06:38.686844 21969 logging.cc:436] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/maintenance_manager.cc:346: suppressed but not reported on 91 messages since previous log ~7 seconds ago