[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251027 03:07:41.950749 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:35899,127.27.151.125:39543,127.27.151.124:46759
I20251027 03:07:41.951064 28253 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 03:07:41.951200 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:41.954905 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:41.954973 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:41.955068 28261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:41.954909 28258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:41.955272 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:41.955315 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:41.955333 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534461955333 us; error 0 us; skew 500 ppm
I20251027 03:07:41.956226 28253 webserver.cc:492] Webserver started at http://127.27.151.126:39201/ using document root <none> and password file <none>
I20251027 03:07:41.956338 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:41.956373 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:41.956424 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:41.957350 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "558bf5d64b34446795b0d26dcfd674f7"
format_stamp: "Formatted at 2025-10-27 03:07:41 on dist-test-slave-4cc8"
I20251027 03:07:41.958361 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251027 03:07:41.958784 28266 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:41.958928 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:41.958985 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "558bf5d64b34446795b0d26dcfd674f7"
format_stamp: "Formatted at 2025-10-27 03:07:41 on dist-test-slave-4cc8"
I20251027 03:07:41.959035 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:41.969017 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:41.969218 28253 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 03:07:41.969308 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:41.972944 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:35899
I20251027 03:07:41.972978 28318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:35899 every 8 connection(s)
I20251027 03:07:41.973312 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:07:41.973461 28319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 03:07:41.974170 28321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:41.974232 28324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:41.974406 28322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:41.974586 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:41.974677 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:41.974709 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:41.974732 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534461974732 us; error 0 us; skew 500 ppm
I20251027 03:07:41.975267 28253 webserver.cc:492] Webserver started at http://127.27.151.125:35797/ using document root <none> and password file <none>
I20251027 03:07:41.975344 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:41.975386 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:41.975426 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:41.975647 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root/instance:
uuid: "b708a058e9f4460fb7c77bca81da0a8c"
format_stamp: "Formatted at 2025-10-27 03:07:41 on dist-test-slave-4cc8"
I20251027 03:07:41.976047 28319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:41.976660 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:41.977151 28331 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:41.977283 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:41.977322 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root
uuid: "b708a058e9f4460fb7c77bca81da0a8c"
format_stamp: "Formatted at 2025-10-27 03:07:41 on dist-test-slave-4cc8"
I20251027 03:07:41.977363 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:41.980655 28319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:41.980942 28267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.125:39543: connect: Connection refused (error 111)
W20251027 03:07:41.981279 28319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.151.125:39543: Network error: Client connection negotiation failed: client connection to 127.27.151.125:39543: connect: Connection refused (error 111)
I20251027 03:07:41.989677 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:41.989890 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:41.993112 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.125:39543
I20251027 03:07:41.993167 28384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.125:39543 every 8 connection(s)
I20251027 03:07:41.993413 28385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:41.993440 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:07:41.994292 28385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:41.994520 28387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:41.995380 28388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:41.995599 28390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:41.995648 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:41.995807 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:41.995846 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:41.995863 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534461995863 us; error 0 us; skew 500 ppm
I20251027 03:07:41.996383 28253 webserver.cc:492] Webserver started at http://127.27.151.124:39541/ using document root <none> and password file <none>
I20251027 03:07:41.996450 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:41.996498 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:41.996548 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:41.996821 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root/instance:
uuid: "b7c63e0493854c88a2dd19f59dff9952"
format_stamp: "Formatted at 2025-10-27 03:07:41 on dist-test-slave-4cc8"
I20251027 03:07:41.997702 28385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:41.998015 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:41.998587 28397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:41.998720 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:41.998769 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root
uuid: "b7c63e0493854c88a2dd19f59dff9952"
format_stamp: "Formatted at 2025-10-27 03:07:41 on dist-test-slave-4cc8"
I20251027 03:07:41.998807 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:42.000888 28385 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:42.001405 28385 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.151.124:46759: Network error: Client connection negotiation failed: client connection to 127.27.151.124:46759: connect: Connection refused (error 111)
I20251027 03:07:42.007992 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:42.008231 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:42.011278 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.124:46759
I20251027 03:07:42.011327 28450 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.124:46759 every 8 connection(s)
I20251027 03:07:42.011471 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:07:42.011569 28451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:42.012091 28451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:42.014624 28451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:42.016811 28451 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:42.020882 28451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: Bootstrap starting.
I20251027 03:07:42.021627 28451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:42.021840 28451 log.cc:826] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: Log is configured to *not* fsync() on all Append() calls
I20251027 03:07:42.022300 28451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: No bootstrap required, opened a new log
I20251027 03:07:42.023347 28451 raft_consensus.cc:359] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.023463 28451 raft_consensus.cc:385] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:42.023491 28451 raft_consensus.cc:740] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c63e0493854c88a2dd19f59dff9952, State: Initialized, Role: FOLLOWER
I20251027 03:07:42.023576 28451 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.023779 28456 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.023836 28451 sys_catalog.cc:565] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:42.023851 28456 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: This master's current role is: FOLLOWER
W20251027 03:07:42.025303 28467 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 03:07:42.025344 28467 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 03:07:42.030493 28319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } attempt: 1
I20251027 03:07:42.033214 28319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:42.035975 28319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7: Bootstrap starting.
I20251027 03:07:42.036381 28319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:42.037046 28319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7: No bootstrap required, opened a new log
I20251027 03:07:42.037186 28319 raft_consensus.cc:359] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.037232 28319 raft_consensus.cc:385] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:42.037246 28319 raft_consensus.cc:740] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558bf5d64b34446795b0d26dcfd674f7, State: Initialized, Role: FOLLOWER
I20251027 03:07:42.037281 28319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.037458 28319 sys_catalog.cc:565] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:42.037451 28470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.038645 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20251027 03:07:42.038761 28470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: This master's current role is: FOLLOWER
W20251027 03:07:42.038771 28481 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 03:07:42.038877 28481 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 03:07:42.053534 28385 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } attempt: 1
I20251027 03:07:42.056831 28385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: Bootstrap starting.
I20251027 03:07:42.057197 28385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:42.057709 28385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: No bootstrap required, opened a new log
I20251027 03:07:42.057844 28385 raft_consensus.cc:359] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.057891 28385 raft_consensus.cc:385] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:42.057904 28385 raft_consensus.cc:740] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b708a058e9f4460fb7c77bca81da0a8c, State: Initialized, Role: FOLLOWER
I20251027 03:07:42.057940 28385 consensus_queue.cc:260] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.058156 28385 sys_catalog.cc:565] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:42.058167 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.058283 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:42.059391 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20251027 03:07:42.059675 28494 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 03:07:42.059715 28494 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 03:07:42.059700 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:42.061097 28495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:42.061097 28496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:42.064882 28498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:42.065016 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:42.065122 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:42.065156 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:42.065183 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534462065183 us; error 0 us; skew 500 ppm
I20251027 03:07:42.065734 28253 webserver.cc:492] Webserver started at http://127.27.151.65:46725/ using document root <none> and password file <none>
I20251027 03:07:42.065814 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:42.065862 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:42.065907 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:42.066143 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "908a727a78e74ee8941ed8e9d4ae32ce"
format_stamp: "Formatted at 2025-10-27 03:07:42 on dist-test-slave-4cc8"
I20251027 03:07:42.066957 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:42.067502 28503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.067607 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.067646 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "908a727a78e74ee8941ed8e9d4ae32ce"
format_stamp: "Formatted at 2025-10-27 03:07:42 on dist-test-slave-4cc8"
I20251027 03:07:42.067689 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:42.071805 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:42.072027 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:42.072219 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:42.072499 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:42.072533 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.072564 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:42.072609 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.076671 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:45483
I20251027 03:07:42.076961 28566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:45483 every 8 connection(s)
I20251027 03:07:42.077257 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:42.079648 28573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:42.080629 28567 heartbeater.cc:344] Connected to a master server at 127.27.151.124:46759
I20251027 03:07:42.080716 28567 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.080964 28567 heartbeater.cc:507] Master 127.27.151.124:46759 requested a full tablet report, sending...
I20251027 03:07:42.081610 28415 ts_manager.cc:194] Registered new tserver with Master: 908a727a78e74ee8941ed8e9d4ae32ce (127.27.151.65:45483)
I20251027 03:07:42.081954 28569 heartbeater.cc:344] Connected to a master server at 127.27.151.125:39543
I20251027 03:07:42.082057 28569 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.082088 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:42.082221 28569 heartbeater.cc:507] Master 127.27.151.125:39543 requested a full tablet report, sending...
W20251027 03:07:42.082237 28577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:42.082083 28575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:42.082450 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:42.082489 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:42.082494 28349 ts_manager.cc:194] Registered new tserver with Master: 908a727a78e74ee8941ed8e9d4ae32ce (127.27.151.65:45483)
I20251027 03:07:42.082525 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534462082525 us; error 0 us; skew 500 ppm
I20251027 03:07:42.083052 28568 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35899
I20251027 03:07:42.083140 28253 webserver.cc:492] Webserver started at http://127.27.151.66:44377/ using document root <none> and password file <none>
I20251027 03:07:42.083153 28568 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.083232 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:42.083268 28568 heartbeater.cc:507] Master 127.27.151.126:35899 requested a full tablet report, sending...
I20251027 03:07:42.083276 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:42.083339 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:42.083500 28283 ts_manager.cc:194] Registered new tserver with Master: 908a727a78e74ee8941ed8e9d4ae32ce (127.27.151.65:45483)
I20251027 03:07:42.083580 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "692eedefe5534a64a0e445fc374d584e"
format_stamp: "Formatted at 2025-10-27 03:07:42 on dist-test-slave-4cc8"
I20251027 03:07:42.084443 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:42.084939 28582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.085042 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.085084 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "692eedefe5534a64a0e445fc374d584e"
format_stamp: "Formatted at 2025-10-27 03:07:42 on dist-test-slave-4cc8"
I20251027 03:07:42.085126 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:42.103606 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:42.103765 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:42.103904 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:42.104069 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:42.104103 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.104121 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:42.104135 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.108124 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:40945
I20251027 03:07:42.108187 28645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:40945 every 8 connection(s)
I20251027 03:07:42.108579 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:07:42.110577 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:42.110780 28653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:42.110848 28652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:42.110930 28655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:42.111099 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:42.111135 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:42.111150 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534462111151 us; error 0 us; skew 500 ppm
I20251027 03:07:42.111780 28253 webserver.cc:492] Webserver started at http://127.27.151.67:42653/ using document root <none> and password file <none>
I20251027 03:07:42.112166 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:42.112211 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:42.112249 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:42.112514 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "368a232b891045cfac26a9db96d38f4f"
format_stamp: "Formatted at 2025-10-27 03:07:42 on dist-test-slave-4cc8"
I20251027 03:07:42.112732 28646 heartbeater.cc:344] Connected to a master server at 127.27.151.124:46759
I20251027 03:07:42.112774 28646 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.112877 28646 heartbeater.cc:507] Master 127.27.151.124:46759 requested a full tablet report, sending...
I20251027 03:07:42.113099 28415 ts_manager.cc:194] Registered new tserver with Master: 692eedefe5534a64a0e445fc374d584e (127.27.151.66:40945)
I20251027 03:07:42.113485 28648 heartbeater.cc:344] Connected to a master server at 127.27.151.125:39543
I20251027 03:07:42.113575 28648 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.113698 28648 heartbeater.cc:507] Master 127.27.151.125:39543 requested a full tablet report, sending...
I20251027 03:07:42.114358 28349 ts_manager.cc:194] Registered new tserver with Master: 692eedefe5534a64a0e445fc374d584e (127.27.151.66:40945)
I20251027 03:07:42.114758 28647 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35899
I20251027 03:07:42.114800 28647 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.114892 28647 heartbeater.cc:507] Master 127.27.151.126:35899 requested a full tablet report, sending...
I20251027 03:07:42.115128 28283 ts_manager.cc:194] Registered new tserver with Master: 692eedefe5534a64a0e445fc374d584e (127.27.151.66:40945)
I20251027 03:07:42.116308 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20251027 03:07:42.116747 28661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.116881 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.116925 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "368a232b891045cfac26a9db96d38f4f"
format_stamp: "Formatted at 2025-10-27 03:07:42 on dist-test-slave-4cc8"
I20251027 03:07:42.116969 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:42.141072 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:42.141562 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:42.141695 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:42.141898 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:42.141927 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.141969 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:42.141989 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:42.145277 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:39901
I20251027 03:07:42.145418 28724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:39901 every 8 connection(s)
I20251027 03:07:42.148864 28725 heartbeater.cc:344] Connected to a master server at 127.27.151.124:46759
I20251027 03:07:42.148917 28725 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.149010 28725 heartbeater.cc:507] Master 127.27.151.124:46759 requested a full tablet report, sending...
I20251027 03:07:42.149251 28415 ts_manager.cc:194] Registered new tserver with Master: 368a232b891045cfac26a9db96d38f4f (127.27.151.67:39901)
I20251027 03:07:42.149490 28726 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35899
I20251027 03:07:42.149538 28726 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.149621 28726 heartbeater.cc:507] Master 127.27.151.126:35899 requested a full tablet report, sending...
I20251027 03:07:42.149626 28727 heartbeater.cc:344] Connected to a master server at 127.27.151.125:39543
I20251027 03:07:42.149693 28727 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:42.149783 28727 heartbeater.cc:507] Master 127.27.151.125:39543 requested a full tablet report, sending...
I20251027 03:07:42.149843 28283 ts_manager.cc:194] Registered new tserver with Master: 368a232b891045cfac26a9db96d38f4f (127.27.151.67:39901)
I20251027 03:07:42.150014 28349 ts_manager.cc:194] Registered new tserver with Master: 368a232b891045cfac26a9db96d38f4f (127.27.151.67:39901)
I20251027 03:07:42.150487 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00492596s
I20251027 03:07:42.189199 28470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:42.189301 28470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.189529 28470 leader_election.cc:290] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b708a058e9f4460fb7c77bca81da0a8c (127.27.151.125:39543), b7c63e0493854c88a2dd19f59dff9952 (127.27.151.124:46759)
I20251027 03:07:42.189747 28425 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "558bf5d64b34446795b0d26dcfd674f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c63e0493854c88a2dd19f59dff9952" is_pre_election: true
I20251027 03:07:42.189747 28359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "558bf5d64b34446795b0d26dcfd674f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b708a058e9f4460fb7c77bca81da0a8c" is_pre_election: true
I20251027 03:07:42.189889 28359 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 558bf5d64b34446795b0d26dcfd674f7 in term 0.
I20251027 03:07:42.189889 28425 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 558bf5d64b34446795b0d26dcfd674f7 in term 0.
I20251027 03:07:42.190032 28270 leader_election.cc:304] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 558bf5d64b34446795b0d26dcfd674f7, b7c63e0493854c88a2dd19f59dff9952; no voters:
I20251027 03:07:42.190146 28470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:42.190189 28470 raft_consensus.cc:493] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:42.190219 28470 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:42.190851 28470 raft_consensus.cc:515] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.190986 28470 leader_election.cc:290] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [CANDIDATE]: Term 1 election: Requested vote from peers b708a058e9f4460fb7c77bca81da0a8c (127.27.151.125:39543), b7c63e0493854c88a2dd19f59dff9952 (127.27.151.124:46759)
I20251027 03:07:42.191121 28425 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "558bf5d64b34446795b0d26dcfd674f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7c63e0493854c88a2dd19f59dff9952"
I20251027 03:07:42.191180 28425 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:42.191155 28359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "558bf5d64b34446795b0d26dcfd674f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b708a058e9f4460fb7c77bca81da0a8c"
I20251027 03:07:42.191221 28359 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:42.191669 28425 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558bf5d64b34446795b0d26dcfd674f7 in term 1.
I20251027 03:07:42.191823 28359 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 558bf5d64b34446795b0d26dcfd674f7 in term 1.
I20251027 03:07:42.191845 28270 leader_election.cc:304] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 558bf5d64b34446795b0d26dcfd674f7, b7c63e0493854c88a2dd19f59dff9952; no voters:
I20251027 03:07:42.191998 28470 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:42.192173 28470 raft_consensus.cc:697] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 1 LEADER]: Becoming Leader. State: Replica: 558bf5d64b34446795b0d26dcfd674f7, State: Running, Role: LEADER
I20251027 03:07:42.192252 28470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } }
I20251027 03:07:42.192615 28733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558bf5d64b34446795b0d26dcfd674f7. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.192698 28733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:42.192914 28735 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:07:42.193084 28735 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:07:42.194118 28359 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 1 FOLLOWER]: Refusing update from remote peer 558bf5d64b34446795b0d26dcfd674f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:42.194118 28425 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 1 FOLLOWER]: Refusing update from remote peer 558bf5d64b34446795b0d26dcfd674f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:42.194334 28470 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:42.194403 28733 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:42.195082 28456 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558bf5d64b34446795b0d26dcfd674f7. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.195083 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558bf5d64b34446795b0d26dcfd674f7. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.195185 28456 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:42.195194 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:42.195672 28733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.195742 28470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.195798 28470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:42.195745 28733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:42.195816 28456 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.195889 28456 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:42.195874 28483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "558bf5d64b34446795b0d26dcfd674f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "558bf5d64b34446795b0d26dcfd674f7" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 35899 } } peers { permanent_uuid: "b708a058e9f4460fb7c77bca81da0a8c" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 39543 } } peers { permanent_uuid: "b7c63e0493854c88a2dd19f59dff9952" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 46759 } } }
I20251027 03:07:42.195926 28483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:42.195955 28736 mvcc.cc:204] Tried to move back new op lower bound from 7215245157145632768 to 7215245157140160512. Current Snapshot: MvccSnapshot[applied={T|T < 7215245157145632768}]
I20251027 03:07:42.196244 28735 catalog_manager.cc:1357] Generated new cluster ID: daf6274a9afa450f8cddee0c10b04b38
I20251027 03:07:42.196286 28735 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:07:42.206339 28735 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:07:42.206943 28735 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:07:42.215076 28735 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7: Generated new TSK 0
I20251027 03:07:42.215229 28735 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:07:43.025612 28467 catalog_manager.cc:1269] Loaded cluster ID: daf6274a9afa450f8cddee0c10b04b38
I20251027 03:07:43.025679 28467 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: loading cluster ID for follower catalog manager: success
I20251027 03:07:43.026367 28467 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: acquiring CA information for follower catalog manager: success
I20251027 03:07:43.026531 28467 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 03:07:43.059965 28494 catalog_manager.cc:1269] Loaded cluster ID: daf6274a9afa450f8cddee0c10b04b38
I20251027 03:07:43.060032 28494 catalog_manager.cc:1562] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: loading cluster ID for follower catalog manager: success
I20251027 03:07:43.060652 28494 catalog_manager.cc:1584] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: acquiring CA information for follower catalog manager: success
I20251027 03:07:43.060745 28494 catalog_manager.cc:1612] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 03:07:43.084596 28283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58278
I20251027 03:07:43.115876 28283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58284
I20251027 03:07:43.150734 28283 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58298
I20251027 03:07:44.038669 28480 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:07:44.085208 28568 heartbeater.cc:499] Master 127.27.151.126:35899 was elected leader, sending a full tablet report...
I20251027 03:07:44.116371 28647 heartbeater.cc:499] Master 127.27.151.126:35899 was elected leader, sending a full tablet report...
I20251027 03:07:44.151181 28726 heartbeater.cc:499] Master 127.27.151.126:35899 was elected leader, sending a full tablet report...
I20251027 03:07:44.300156 28253 test_util.cc:276] Using random seed: 872485992
I20251027 03:07:44.307498 28283 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58312:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 03:07:44.314253 28689 tablet_service.cc:1505] Processing CreateTablet for tablet 4b7644c0f2b940f58392ce6c64cba8cd (DEFAULT_TABLE table=test-workload [id=f9df79747ecb4877946755ae45b8287d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:07:44.314250 28610 tablet_service.cc:1505] Processing CreateTablet for tablet b6de8b4d6d26438484b59624b5d2531d (DEFAULT_TABLE table=test-workload [id=f9df79747ecb4877946755ae45b8287d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:07:44.314481 28531 tablet_service.cc:1505] Processing CreateTablet for tablet bbf95d6b7a554a39a37159b89123d9ab (DEFAULT_TABLE table=test-workload [id=f9df79747ecb4877946755ae45b8287d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:07:44.314505 28689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b7644c0f2b940f58392ce6c64cba8cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:44.314558 28610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6de8b4d6d26438484b59624b5d2531d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:44.314616 28530 tablet_service.cc:1505] Processing CreateTablet for tablet 0aefe43a98a542a8a57bdd3c71f7c27b (DEFAULT_TABLE table=test-workload [id=f9df79747ecb4877946755ae45b8287d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:07:44.315745 28773 tablet_bootstrap.cc:492] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f: Bootstrap starting.
I20251027 03:07:44.316025 28774 tablet_bootstrap.cc:492] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e: Bootstrap starting.
I20251027 03:07:44.314569 28531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbf95d6b7a554a39a37159b89123d9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:44.316200 28773 tablet_bootstrap.cc:654] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:44.316285 28530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aefe43a98a542a8a57bdd3c71f7c27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:44.316399 28774 tablet_bootstrap.cc:654] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:44.316942 28774 tablet_bootstrap.cc:492] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e: No bootstrap required, opened a new log
I20251027 03:07:44.316980 28774 ts_tablet_manager.cc:1403] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:44.317118 28774 raft_consensus.cc:359] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692eedefe5534a64a0e445fc374d584e" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40945 } }
I20251027 03:07:44.317167 28774 raft_consensus.cc:385] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:44.317188 28774 raft_consensus.cc:740] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692eedefe5534a64a0e445fc374d584e, State: Initialized, Role: FOLLOWER
I20251027 03:07:44.317252 28774 consensus_queue.cc:260] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692eedefe5534a64a0e445fc374d584e" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40945 } }
I20251027 03:07:44.317291 28774 raft_consensus.cc:399] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:44.317312 28774 raft_consensus.cc:493] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:44.317332 28774 raft_consensus.cc:3060] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:44.317387 28777 tablet_bootstrap.cc:492] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: Bootstrap starting.
I20251027 03:07:44.317739 28777 tablet_bootstrap.cc:654] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:44.317800 28773 tablet_bootstrap.cc:492] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f: No bootstrap required, opened a new log
I20251027 03:07:44.317869 28773 ts_tablet_manager.cc:1403] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:44.318034 28774 raft_consensus.cc:515] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692eedefe5534a64a0e445fc374d584e" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40945 } }
I20251027 03:07:44.318046 28773 raft_consensus.cc:359] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a232b891045cfac26a9db96d38f4f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39901 } }
I20251027 03:07:44.318145 28774 leader_election.cc:304] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692eedefe5534a64a0e445fc374d584e; no voters:
I20251027 03:07:44.318174 28773 raft_consensus.cc:385] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:44.318230 28773 raft_consensus.cc:740] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368a232b891045cfac26a9db96d38f4f, State: Initialized, Role: FOLLOWER
I20251027 03:07:44.318267 28774 leader_election.cc:290] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:44.318281 28773 consensus_queue.cc:260] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a232b891045cfac26a9db96d38f4f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39901 } }
I20251027 03:07:44.318336 28773 raft_consensus.cc:399] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:44.318359 28773 raft_consensus.cc:493] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:44.318387 28773 raft_consensus.cc:3060] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:44.318397 28774 ts_tablet_manager.cc:1434] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251027 03:07:44.318406 28777 tablet_bootstrap.cc:492] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: No bootstrap required, opened a new log
I20251027 03:07:44.318464 28777 ts_tablet_manager.cc:1403] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:44.318571 28777 raft_consensus.cc:359] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.318610 28777 raft_consensus.cc:385] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:44.318627 28777 raft_consensus.cc:740] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908a727a78e74ee8941ed8e9d4ae32ce, State: Initialized, Role: FOLLOWER
I20251027 03:07:44.318668 28777 consensus_queue.cc:260] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.318714 28777 raft_consensus.cc:399] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:44.318735 28777 raft_consensus.cc:493] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:44.318754 28777 raft_consensus.cc:3060] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:44.318964 28773 raft_consensus.cc:515] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a232b891045cfac26a9db96d38f4f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39901 } }
I20251027 03:07:44.319011 28773 leader_election.cc:304] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 368a232b891045cfac26a9db96d38f4f; no voters:
I20251027 03:07:44.319100 28773 leader_election.cc:290] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:44.319157 28781 raft_consensus.cc:2804] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:44.319178 28773 ts_tablet_manager.cc:1434] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:44.319228 28777 raft_consensus.cc:515] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.319286 28777 leader_election.cc:304] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 908a727a78e74ee8941ed8e9d4ae32ce; no voters:
I20251027 03:07:44.319290 28781 raft_consensus.cc:697] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 1 LEADER]: Becoming Leader. State: Replica: 368a232b891045cfac26a9db96d38f4f, State: Running, Role: LEADER
I20251027 03:07:44.319398 28781 consensus_queue.cc:237] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a232b891045cfac26a9db96d38f4f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39901 } }
I20251027 03:07:44.319398 28777 leader_election.cc:290] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:44.319439 28784 raft_consensus.cc:2804] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:44.319576 28784 raft_consensus.cc:697] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 LEADER]: Becoming Leader. State: Replica: 908a727a78e74ee8941ed8e9d4ae32ce, State: Running, Role: LEADER
I20251027 03:07:44.319599 28777 ts_tablet_manager.cc:1434] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:44.319674 28777 tablet_bootstrap.cc:492] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce: Bootstrap starting.
I20251027 03:07:44.319615 28784 consensus_queue.cc:237] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.319932 28777 tablet_bootstrap.cc:654] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:44.320448 28283 catalog_manager.cc:5649] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 908a727a78e74ee8941ed8e9d4ae32ce (127.27.151.65). New cstate: current_term: 1 leader_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:44.320585 28282 catalog_manager.cc:5649] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 368a232b891045cfac26a9db96d38f4f (127.27.151.67). New cstate: current_term: 1 leader_uuid: "368a232b891045cfac26a9db96d38f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368a232b891045cfac26a9db96d38f4f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39901 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:44.321233 28779 raft_consensus.cc:2804] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:44.321395 28779 raft_consensus.cc:697] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 1 LEADER]: Becoming Leader. State: Replica: 692eedefe5534a64a0e445fc374d584e, State: Running, Role: LEADER
I20251027 03:07:44.321494 28779 consensus_queue.cc:237] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692eedefe5534a64a0e445fc374d584e" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40945 } }
I20251027 03:07:44.322054 28777 tablet_bootstrap.cc:492] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce: No bootstrap required, opened a new log
I20251027 03:07:44.322114 28777 ts_tablet_manager.cc:1403] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:44.322211 28777 raft_consensus.cc:359] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.322268 28777 raft_consensus.cc:385] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:44.322213 28282 catalog_manager.cc:5649] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e reported cstate change: term changed from 0 to 1, leader changed from <none> to 692eedefe5534a64a0e445fc374d584e (127.27.151.66). New cstate: current_term: 1 leader_uuid: "692eedefe5534a64a0e445fc374d584e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692eedefe5534a64a0e445fc374d584e" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40945 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:44.322291 28777 raft_consensus.cc:740] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908a727a78e74ee8941ed8e9d4ae32ce, State: Initialized, Role: FOLLOWER
I20251027 03:07:44.322501 28777 consensus_queue.cc:260] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.322554 28777 raft_consensus.cc:399] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:44.322580 28777 raft_consensus.cc:493] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:44.322605 28777 raft_consensus.cc:3060] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:44.323256 28777 raft_consensus.cc:515] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.323313 28777 leader_election.cc:304] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 908a727a78e74ee8941ed8e9d4ae32ce; no voters:
I20251027 03:07:44.323356 28777 leader_election.cc:290] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:44.323380 28785 raft_consensus.cc:2804] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:44.323419 28777 ts_tablet_manager.cc:1434] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:07:44.323436 28785 raft_consensus.cc:697] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 LEADER]: Becoming Leader. State: Replica: 908a727a78e74ee8941ed8e9d4ae32ce, State: Running, Role: LEADER
I20251027 03:07:44.323472 28785 consensus_queue.cc:237] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } }
I20251027 03:07:44.324281 28283 catalog_manager.cc:5649] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 908a727a78e74ee8941ed8e9d4ae32ce (127.27.151.65). New cstate: current_term: 1 leader_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45483 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:44.326272 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:07:44.328797 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:44.328969 28253 tablet_replica.cc:333] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce: stopping tablet replica
I20251027 03:07:44.329056 28253 raft_consensus.cc:2243] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:44.329097 28253 raft_consensus.cc:2272] T bbf95d6b7a554a39a37159b89123d9ab P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:44.329366 28253 tablet_replica.cc:333] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: stopping tablet replica
I20251027 03:07:44.329404 28253 raft_consensus.cc:2243] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:44.329427 28253 pending_rounds.cc:70] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: Trying to abort 1 pending ops.
I20251027 03:07:44.329447 28253 pending_rounds.cc:77] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245165853728768 op_type: NO_OP noop_request { }
I20251027 03:07:44.329481 28253 raft_consensus.cc:2889] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:07:44.329505 28253 raft_consensus.cc:2272] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 03:07:44.336313 28788 consensus_queue.cc:1175] T 0aefe43a98a542a8a57bdd3c71f7c27b P 908a727a78e74ee8941ed8e9d4ae32ce [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "908a727a78e74ee8941ed8e9d4ae32ce" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251027 03:07:44.348349 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:07:44.349282 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:07:44.352394 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:44.352557 28253 tablet_replica.cc:333] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e: stopping tablet replica
I20251027 03:07:44.352631 28253 raft_consensus.cc:2243] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:44.352677 28253 raft_consensus.cc:2272] T b6de8b4d6d26438484b59624b5d2531d P 692eedefe5534a64a0e445fc374d584e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:44.354064 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:07:44.354719 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:07:44.356990 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:44.357141 28253 tablet_replica.cc:333] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f: stopping tablet replica
I20251027 03:07:44.357198 28253 raft_consensus.cc:2243] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:44.357237 28253 raft_consensus.cc:2272] T 4b7644c0f2b940f58392ce6c64cba8cd P 368a232b891045cfac26a9db96d38f4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:44.368814 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:07:44.369556 28253 master.cc:561] Master@127.27.151.126:35899 shutting down...
I20251027 03:07:44.371618 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:44.371716 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:44.371753 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 558bf5d64b34446795b0d26dcfd674f7: stopping tablet replica
I20251027 03:07:44.373788 28253 master.cc:583] Master@127.27.151.126:35899 shutdown complete.
I20251027 03:07:44.374701 28253 master.cc:561] Master@127.27.151.125:39543 shutting down...
I20251027 03:07:44.377261 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:44.377311 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:44.377341 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P b708a058e9f4460fb7c77bca81da0a8c: stopping tablet replica
I20251027 03:07:44.388906 28253 master.cc:583] Master@127.27.151.125:39543 shutdown complete.
I20251027 03:07:44.389856 28253 master.cc:561] Master@127.27.151.124:46759 shutting down...
I20251027 03:07:44.392433 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:44.392506 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:44.392544 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P b7c63e0493854c88a2dd19f59dff9952: stopping tablet replica
I20251027 03:07:44.404256 28253 master.cc:583] Master@127.27.151.124:46759 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2458 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20251027 03:07:44.409291 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:42585
I20251027 03:07:44.409408 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:44.410383 28801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.410483 28800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.410521 28803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:44.410671 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:44.410765 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:44.410797 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:44.410810 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534464410811 us; error 0 us; skew 500 ppm
I20251027 03:07:44.411298 28253 webserver.cc:492] Webserver started at http://127.27.151.126:39249/ using document root <none> and password file <none>
I20251027 03:07:44.411371 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:44.411417 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:44.411461 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:44.411684 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "ef229b83cee04555b0e559ac37fb88a0"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.412561 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:44.412983 28808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.413127 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.413168 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "ef229b83cee04555b0e559ac37fb88a0"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.413218 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:44.422441 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:44.422590 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:44.424863 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:42585
I20251027 03:07:44.427721 28860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:42585 every 8 connection(s)
I20251027 03:07:44.427799 28861 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:44.428833 28861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0: Bootstrap starting.
I20251027 03:07:44.429054 28861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:44.429450 28861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0: No bootstrap required, opened a new log
I20251027 03:07:44.429538 28861 raft_consensus.cc:359] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef229b83cee04555b0e559ac37fb88a0" member_type: VOTER }
I20251027 03:07:44.429577 28861 raft_consensus.cc:385] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:44.429591 28861 raft_consensus.cc:740] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef229b83cee04555b0e559ac37fb88a0, State: Initialized, Role: FOLLOWER
I20251027 03:07:44.429626 28861 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef229b83cee04555b0e559ac37fb88a0" member_type: VOTER }
I20251027 03:07:44.429670 28861 raft_consensus.cc:399] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:44.429698 28861 raft_consensus.cc:493] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:44.429718 28861 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:44.430132 28861 raft_consensus.cc:515] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef229b83cee04555b0e559ac37fb88a0" member_type: VOTER }
I20251027 03:07:44.430181 28861 leader_election.cc:304] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef229b83cee04555b0e559ac37fb88a0; no voters:
I20251027 03:07:44.430259 28861 leader_election.cc:290] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:44.430316 28864 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:44.430368 28861 sys_catalog.cc:565] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:44.430436 28864 raft_consensus.cc:697] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 1 LEADER]: Becoming Leader. State: Replica: ef229b83cee04555b0e559ac37fb88a0, State: Running, Role: LEADER
I20251027 03:07:44.430483 28864 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef229b83cee04555b0e559ac37fb88a0" member_type: VOTER }
I20251027 03:07:44.430634 28864 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef229b83cee04555b0e559ac37fb88a0. Latest consensus state: current_term: 1 leader_uuid: "ef229b83cee04555b0e559ac37fb88a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef229b83cee04555b0e559ac37fb88a0" member_type: VOTER } }
I20251027 03:07:44.430689 28864 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:44.430660 28866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef229b83cee04555b0e559ac37fb88a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef229b83cee04555b0e559ac37fb88a0" member_type: VOTER } }
I20251027 03:07:44.430858 28866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:44.430997 28869 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:07:44.431125 28869 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:07:44.431627 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:07:44.431631 28869 catalog_manager.cc:1357] Generated new cluster ID: ff24642e65834ca1b120c505759d5dda
I20251027 03:07:44.431690 28869 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:07:44.446020 28869 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:07:44.446404 28869 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:07:44.453033 28869 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0: Generated new TSK 0
I20251027 03:07:44.453101 28869 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:07:44.463258 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:44.464298 28882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.464362 28883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.464394 28885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:44.464509 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:44.464591 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:44.464624 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:44.464641 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534464464642 us; error 0 us; skew 500 ppm
I20251027 03:07:44.465150 28253 webserver.cc:492] Webserver started at http://127.27.151.65:39171/ using document root <none> and password file <none>
I20251027 03:07:44.465224 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:44.465269 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:44.465317 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:44.465540 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "9652d571953345188c5ccaab47629bec"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.466342 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:44.466720 28890 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.466810 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.466863 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "9652d571953345188c5ccaab47629bec"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.466900 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:44.476517 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:44.476691 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:44.476805 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:44.476974 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:44.476998 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.477017 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:44.477043 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.479532 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:41377
I20251027 03:07:44.479558 28953 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:41377 every 8 connection(s)
I20251027 03:07:44.479809 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:44.480877 28961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.480940 28958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:44.481225 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:44.481426 28959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:44.481583 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:44.481614 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:44.481626 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534464481627 us; error 0 us; skew 500 ppm
I20251027 03:07:44.482154 28253 webserver.cc:492] Webserver started at http://127.27.151.66:33459/ using document root <none> and password file <none>
I20251027 03:07:44.482232 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:44.482275 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:44.482318 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:44.482579 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "182815b12fa94a2b88c3ddc0b49e4342"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.483444 28954 heartbeater.cc:344] Connected to a master server at 127.27.151.126:42585
I20251027 03:07:44.483500 28954 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:44.483582 28954 heartbeater.cc:507] Master 127.27.151.126:42585 requested a full tablet report, sending...
I20251027 03:07:44.483597 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:44.483788 28825 ts_manager.cc:194] Registered new tserver with Master: 9652d571953345188c5ccaab47629bec (127.27.151.65:41377)
I20251027 03:07:44.484055 28966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.484211 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.484261 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "182815b12fa94a2b88c3ddc0b49e4342"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.484292 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:44.484531 28825 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36610
I20251027 03:07:44.498186 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:44.498332 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:44.498420 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:44.498586 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:44.498612 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.498641 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:44.498667 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.501412 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:37695
I20251027 03:07:44.501452 29029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:37695 every 8 connection(s)
I20251027 03:07:44.501718 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:44.502743 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.502806 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:44.502987 29036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:44.503010 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:44.503190 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:44.503221 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:44.503232 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534464503232 us; error 0 us; skew 500 ppm
I20251027 03:07:44.503708 28253 webserver.cc:492] Webserver started at http://127.27.151.67:36391/ using document root <none> and password file <none>
I20251027 03:07:44.503788 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:44.503825 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:44.503873 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:44.504123 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "8e1ed5f3b1ac408091382b39763e63b4"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.504535 29030 heartbeater.cc:344] Connected to a master server at 127.27.151.126:42585
I20251027 03:07:44.504594 29030 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:44.504678 29030 heartbeater.cc:507] Master 127.27.151.126:42585 requested a full tablet report, sending...
I20251027 03:07:44.504882 28825 ts_manager.cc:194] Registered new tserver with Master: 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:44.505172 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:44.505223 28825 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36624
I20251027 03:07:44.505652 29041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.505770 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.505815 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "8e1ed5f3b1ac408091382b39763e63b4"
format_stamp: "Formatted at 2025-10-27 03:07:44 on dist-test-slave-4cc8"
I20251027 03:07:44.505858 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:44.513655 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:44.513885 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:44.514019 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:44.514187 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:44.514214 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.514240 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:44.514266 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:44.517225 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:35899
I20251027 03:07:44.517275 29104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:35899 every 8 connection(s)
I20251027 03:07:44.519865 29105 heartbeater.cc:344] Connected to a master server at 127.27.151.126:42585
I20251027 03:07:44.519917 29105 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:44.520006 29105 heartbeater.cc:507] Master 127.27.151.126:42585 requested a full tablet report, sending...
I20251027 03:07:44.520229 28825 ts_manager.cc:194] Registered new tserver with Master: 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:44.520509 28825 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36630
I20251027 03:07:44.520704 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003328501s
I20251027 03:07:45.485121 28954 heartbeater.cc:499] Master 127.27.151.126:42585 was elected leader, sending a full tablet report...
I20251027 03:07:45.505637 29030 heartbeater.cc:499] Master 127.27.151.126:42585 was elected leader, sending a full tablet report...
I20251027 03:07:45.520970 29105 heartbeater.cc:499] Master 127.27.151.126:42585 was elected leader, sending a full tablet report...
I20251027 03:07:45.545504 28253 test_util.cc:276] Using random seed: 873731358
I20251027 03:07:45.553414 28825 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36646:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:07:45.553566 28825 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:07:45.558140 28994 tablet_service.cc:1505] Processing CreateTablet for tablet 2cfff53f915f4279b49cc70c3fd67ee8 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251027 03:07:45.558284 28994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cfff53f915f4279b49cc70c3fd67ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.558761 28987 tablet_service.cc:1505] Processing CreateTablet for tablet 59285374574c4ce6b2a2612a90e60612 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251027 03:07:45.558830 29069 tablet_service.cc:1505] Processing CreateTablet for tablet 2cfff53f915f4279b49cc70c3fd67ee8 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251027 03:07:45.558879 28987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59285374574c4ce6b2a2612a90e60612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.558924 29069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cfff53f915f4279b49cc70c3fd67ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.559582 29068 tablet_service.cc:1505] Processing CreateTablet for tablet 4e2b554f96a148edb80ce6f008a46baa (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251027 03:07:45.559657 29068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2b554f96a148edb80ce6f008a46baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.559862 28988 tablet_service.cc:1505] Processing CreateTablet for tablet 211f82947754441f8f296df38ad1a0ba (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251027 03:07:45.559942 28988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f82947754441f8f296df38ad1a0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.560218 29126 tablet_bootstrap.cc:492] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.560590 29126 tablet_bootstrap.cc:654] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.560814 29067 tablet_service.cc:1505] Processing CreateTablet for tablet a5876093fda44ea8bac93fdba6731a84 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251027 03:07:45.560899 29067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5876093fda44ea8bac93fdba6731a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.561142 28989 tablet_service.cc:1505] Processing CreateTablet for tablet 90885115b0c240c89820cf90b0cf8bad (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251027 03:07:45.561224 29126 tablet_bootstrap.cc:492] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.561290 28990 tablet_service.cc:1505] Processing CreateTablet for tablet dd52ac8fcb3a4558a20b88edc07b0ea1 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251027 03:07:45.561236 28989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90885115b0c240c89820cf90b0cf8bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.561281 29126 ts_tablet_manager.cc:1403] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.561434 28991 tablet_service.cc:1505] Processing CreateTablet for tablet 89c9d44ada384f12a1365f54af14edb3 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251027 03:07:45.561515 28991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c9d44ada384f12a1365f54af14edb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.561568 28992 tablet_service.cc:1505] Processing CreateTablet for tablet a5876093fda44ea8bac93fdba6731a84 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251027 03:07:45.561647 28992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5876093fda44ea8bac93fdba6731a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.562186 29066 tablet_service.cc:1505] Processing CreateTablet for tablet 89c9d44ada384f12a1365f54af14edb3 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251027 03:07:45.562263 29066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c9d44ada384f12a1365f54af14edb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.561508 29126 raft_consensus.cc:359] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.562788 28993 tablet_service.cc:1505] Processing CreateTablet for tablet 4e2b554f96a148edb80ce6f008a46baa (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251027 03:07:45.562819 29126 raft_consensus.cc:385] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.562842 29126 raft_consensus.cc:740] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.562881 28993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2b554f96a148edb80ce6f008a46baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.562922 29126 consensus_queue.cc:260] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.563087 29126 ts_tablet_manager.cc:1434] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 03:07:45.563282 29126 tablet_bootstrap.cc:492] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.563392 29065 tablet_service.cc:1505] Processing CreateTablet for tablet dd52ac8fcb3a4558a20b88edc07b0ea1 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251027 03:07:45.563477 29065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd52ac8fcb3a4558a20b88edc07b0ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.563640 29126 tablet_bootstrap.cc:654] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.564222 29064 tablet_service.cc:1505] Processing CreateTablet for tablet 90885115b0c240c89820cf90b0cf8bad (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251027 03:07:45.564251 29126 tablet_bootstrap.cc:492] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.564299 28990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd52ac8fcb3a4558a20b88edc07b0ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.564363 29063 tablet_service.cc:1505] Processing CreateTablet for tablet 211f82947754441f8f296df38ad1a0ba (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251027 03:07:45.564455 29062 tablet_service.cc:1505] Processing CreateTablet for tablet 59285374574c4ce6b2a2612a90e60612 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251027 03:07:45.564307 29064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90885115b0c240c89820cf90b0cf8bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.564813 29062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59285374574c4ce6b2a2612a90e60612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.565245 29063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f82947754441f8f296df38ad1a0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.566211 28914 tablet_service.cc:1505] Processing CreateTablet for tablet dd52ac8fcb3a4558a20b88edc07b0ea1 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251027 03:07:45.566279 28918 tablet_service.cc:1505] Processing CreateTablet for tablet 2cfff53f915f4279b49cc70c3fd67ee8 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251027 03:07:45.566291 28914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd52ac8fcb3a4558a20b88edc07b0ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.566426 28917 tablet_service.cc:1505] Processing CreateTablet for tablet 4e2b554f96a148edb80ce6f008a46baa (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251027 03:07:45.566504 28917 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2b554f96a148edb80ce6f008a46baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.564298 29126 ts_tablet_manager.cc:1403] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.566774 29126 raft_consensus.cc:359] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.566844 29126 raft_consensus.cc:385] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.566864 29126 raft_consensus.cc:740] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.566900 29126 consensus_queue.cc:260] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.566984 29126 ts_tablet_manager.cc:1434] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.567065 29126 tablet_bootstrap.cc:492] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.567462 29126 tablet_bootstrap.cc:654] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.567485 29130 tablet_bootstrap.cc:492] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.567873 29125 tablet_bootstrap.cc:492] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.567915 28915 tablet_service.cc:1505] Processing CreateTablet for tablet 89c9d44ada384f12a1365f54af14edb3 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251027 03:07:45.568028 28915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89c9d44ada384f12a1365f54af14edb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.568099 29130 tablet_bootstrap.cc:654] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.568336 29125 tablet_bootstrap.cc:654] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.568538 29126 tablet_bootstrap.cc:492] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.568584 29126 ts_tablet_manager.cc:1403] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:45.568712 29126 raft_consensus.cc:359] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.568742 28911 tablet_service.cc:1505] Processing CreateTablet for tablet 59285374574c4ce6b2a2612a90e60612 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251027 03:07:45.568758 29126 raft_consensus.cc:385] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.568830 29126 raft_consensus.cc:740] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.568856 28911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59285374574c4ce6b2a2612a90e60612. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.568881 29126 consensus_queue.cc:260] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.568969 29126 ts_tablet_manager.cc:1434] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.569025 29126 tablet_bootstrap.cc:492] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.569336 29126 tablet_bootstrap.cc:654] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.569761 28912 tablet_service.cc:1505] Processing CreateTablet for tablet 211f82947754441f8f296df38ad1a0ba (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251027 03:07:45.569824 29126 tablet_bootstrap.cc:492] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.569859 29126 ts_tablet_manager.cc:1403] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:45.569871 28912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 211f82947754441f8f296df38ad1a0ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.569977 29126 raft_consensus.cc:359] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.570052 29126 raft_consensus.cc:385] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.570086 29126 raft_consensus.cc:740] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.570120 29125 tablet_bootstrap.cc:492] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.570175 29125 ts_tablet_manager.cc:1403] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:45.570122 29126 consensus_queue.cc:260] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.570286 29126 ts_tablet_manager.cc:1434] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.570345 29126 tablet_bootstrap.cc:492] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.570327 29125 raft_consensus.cc:359] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.570389 29125 raft_consensus.cc:385] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.570412 29125 raft_consensus.cc:740] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.570482 29125 consensus_queue.cc:260] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.570668 29125 ts_tablet_manager.cc:1434] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:07:45.570720 29126 tablet_bootstrap.cc:654] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.571048 29130 tablet_bootstrap.cc:492] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.571054 29125 tablet_bootstrap.cc:492] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.571146 28913 tablet_service.cc:1505] Processing CreateTablet for tablet 90885115b0c240c89820cf90b0cf8bad (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251027 03:07:45.571179 29130 ts_tablet_manager.cc:1403] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251027 03:07:45.571247 28913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90885115b0c240c89820cf90b0cf8bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.571081 29128 raft_consensus.cc:493] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.571352 29128 raft_consensus.cc:515] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.567873 28916 tablet_service.cc:1505] Processing CreateTablet for tablet a5876093fda44ea8bac93fdba6731a84 (DEFAULT_TABLE table=test-workload [id=35f0c5e1e10845529ab7be6988d69247]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251027 03:07:45.571532 28918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cfff53f915f4279b49cc70c3fd67ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.571585 29125 tablet_bootstrap.cc:654] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.571605 28916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5876093fda44ea8bac93fdba6731a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:45.572007 29128 leader_election.cc:290] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.572079 29126 tablet_bootstrap.cc:492] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.572117 29126 ts_tablet_manager.cc:1403] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:45.572252 29126 raft_consensus.cc:359] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.572299 29126 raft_consensus.cc:385] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.572315 29126 raft_consensus.cc:740] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.572360 29126 consensus_queue.cc:260] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.572444 29126 ts_tablet_manager.cc:1434] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.572497 29126 tablet_bootstrap.cc:492] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.572849 29126 tablet_bootstrap.cc:654] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.571494 29130 raft_consensus.cc:359] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.573275 29130 raft_consensus.cc:385] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.573350 29130 raft_consensus.cc:740] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.573468 29130 consensus_queue.cc:260] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.573562 29125 tablet_bootstrap.cc:492] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.573606 29125 ts_tablet_manager.cc:1403] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251027 03:07:45.573740 29125 raft_consensus.cc:359] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.573805 29125 raft_consensus.cc:385] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.573822 29125 raft_consensus.cc:740] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.573848 29125 consensus_queue.cc:260] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.573995 29125 ts_tablet_manager.cc:1434] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.574041 29125 tablet_bootstrap.cc:492] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.574375 29125 tablet_bootstrap.cc:654] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.574534 29126 tablet_bootstrap.cc:492] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.574570 29126 ts_tablet_manager.cc:1403] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:45.574693 29126 raft_consensus.cc:359] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.574748 29126 raft_consensus.cc:385] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.574772 29126 raft_consensus.cc:740] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.573731 29130 ts_tablet_manager.cc:1434] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 03:07:45.574812 29126 consensus_queue.cc:260] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.574879 29125 tablet_bootstrap.cc:492] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.574903 29126 ts_tablet_manager.cc:1434] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.574924 29125 ts_tablet_manager.cc:1403] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:45.574944 29126 tablet_bootstrap.cc:492] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.575042 29130 tablet_bootstrap.cc:492] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.575043 29125 raft_consensus.cc:359] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.575160 29125 raft_consensus.cc:385] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.575181 29125 raft_consensus.cc:740] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.575240 29125 consensus_queue.cc:260] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.575313 29125 ts_tablet_manager.cc:1434] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.575338 29126 tablet_bootstrap.cc:654] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.575363 29125 tablet_bootstrap.cc:492] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.576133 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5876093fda44ea8bac93fdba6731a84" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec" is_pre_election: true
I20251027 03:07:45.576323 29125 tablet_bootstrap.cc:654] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
W20251027 03:07:45.576435 29045 leader_election.cc:343] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9652d571953345188c5ccaab47629bec (127.27.151.65:41377): Illegal state: must be running to vote when last-logged opid is not known
I20251027 03:07:45.576524 29126 tablet_bootstrap.cc:492] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.576558 29126 ts_tablet_manager.cc:1403] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20251027 03:07:45.576678 29126 raft_consensus.cc:359] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.576745 29126 raft_consensus.cc:385] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.576771 29126 raft_consensus.cc:740] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.576829 29125 tablet_bootstrap.cc:492] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.576876 29125 ts_tablet_manager.cc:1403] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:45.577033 29125 raft_consensus.cc:359] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.576838 29126 consensus_queue.cc:260] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.577124 29125 raft_consensus.cc:385] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.577144 29125 raft_consensus.cc:740] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.577180 29126 ts_tablet_manager.cc:1434] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:07:45.577185 29125 consensus_queue.cc:260] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.577244 29126 tablet_bootstrap.cc:492] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4: Bootstrap starting.
I20251027 03:07:45.577279 29125 ts_tablet_manager.cc:1434] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.577337 29125 tablet_bootstrap.cc:492] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.577453 29130 tablet_bootstrap.cc:654] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.577608 29126 tablet_bootstrap.cc:654] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.577688 29125 tablet_bootstrap.cc:654] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.578193 29126 tablet_bootstrap.cc:492] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4: No bootstrap required, opened a new log
I20251027 03:07:45.578238 29126 ts_tablet_manager.cc:1403] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:07:45.578315 29125 tablet_bootstrap.cc:492] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.578366 29125 ts_tablet_manager.cc:1403] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:45.578365 29126 raft_consensus.cc:359] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.578442 29126 raft_consensus.cc:385] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.578464 29126 raft_consensus.cc:740] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.578505 29126 consensus_queue.cc:260] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.578589 29126 ts_tablet_manager.cc:1434] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.578508 29125 raft_consensus.cc:359] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.578689 29125 raft_consensus.cc:385] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.578708 29125 raft_consensus.cc:740] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.578745 29125 consensus_queue.cc:260] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.578835 29125 ts_tablet_manager.cc:1434] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.578886 29125 tablet_bootstrap.cc:492] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.579092 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5876093fda44ea8bac93fdba6731a84" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
W20251027 03:07:45.579253 29045 leader_election.cc:343] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695): Illegal state: must be running to vote when last-logged opid is not known
I20251027 03:07:45.579262 29125 tablet_bootstrap.cc:654] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.579298 29130 tablet_bootstrap.cc:492] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.579316 29045 leader_election.cc:304] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4; no voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec
I20251027 03:07:45.579339 29130 ts_tablet_manager.cc:1403] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251027 03:07:45.579444 29128 raft_consensus.cc:2749] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 03:07:45.579481 29130 raft_consensus.cc:359] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.579542 29130 raft_consensus.cc:385] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.579564 29130 raft_consensus.cc:740] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.579613 29130 consensus_queue.cc:260] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.579691 29130 ts_tablet_manager.cc:1434] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.579771 29130 tablet_bootstrap.cc:492] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.580099 29130 tablet_bootstrap.cc:654] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.580577 29130 tablet_bootstrap.cc:492] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.580577 29125 tablet_bootstrap.cc:492] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.580654 29125 ts_tablet_manager.cc:1403] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:45.580653 29130 ts_tablet_manager.cc:1403] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.580843 29130 raft_consensus.cc:359] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.580844 29125 raft_consensus.cc:359] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.580924 29130 raft_consensus.cc:385] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.580932 29125 raft_consensus.cc:385] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.580961 29130 raft_consensus.cc:740] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.580966 29125 raft_consensus.cc:740] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.581020 29125 consensus_queue.cc:260] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.581025 29130 consensus_queue.cc:260] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.581108 29125 ts_tablet_manager.cc:1434] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.581125 29130 ts_tablet_manager.cc:1434] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.581166 29125 tablet_bootstrap.cc:492] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.581173 29130 tablet_bootstrap.cc:492] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.581593 29125 tablet_bootstrap.cc:654] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.581761 29130 tablet_bootstrap.cc:654] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.582123 29125 tablet_bootstrap.cc:492] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.582167 29125 ts_tablet_manager.cc:1403] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:45.582294 29130 tablet_bootstrap.cc:492] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.582353 29130 ts_tablet_manager.cc:1403] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.582469 29130 raft_consensus.cc:359] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.582523 29130 raft_consensus.cc:385] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.582536 29130 raft_consensus.cc:740] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.582587 29130 consensus_queue.cc:260] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.582679 29130 ts_tablet_manager.cc:1434] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.582731 29130 tablet_bootstrap.cc:492] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.583120 29130 tablet_bootstrap.cc:654] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.582283 29125 raft_consensus.cc:359] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.583428 29125 raft_consensus.cc:385] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.583443 29125 raft_consensus.cc:740] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.583474 29125 consensus_queue.cc:260] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.583580 29125 ts_tablet_manager.cc:1434] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:07:45.583635 29125 tablet_bootstrap.cc:492] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342: Bootstrap starting.
I20251027 03:07:45.583643 29130 tablet_bootstrap.cc:492] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.583716 29130 ts_tablet_manager.cc:1403] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.583865 29130 raft_consensus.cc:359] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.583927 29130 raft_consensus.cc:385] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.583952 29130 raft_consensus.cc:740] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.583994 29130 consensus_queue.cc:260] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.584077 29130 ts_tablet_manager.cc:1434] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.584125 29130 tablet_bootstrap.cc:492] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.584201 29125 tablet_bootstrap.cc:654] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.584537 29130 tablet_bootstrap.cc:654] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.584903 29125 tablet_bootstrap.cc:492] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342: No bootstrap required, opened a new log
I20251027 03:07:45.584944 29125 ts_tablet_manager.cc:1403] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:45.585033 29130 tablet_bootstrap.cc:492] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.585079 29130 ts_tablet_manager.cc:1403] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.585074 29125 raft_consensus.cc:359] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.585155 29125 raft_consensus.cc:385] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.585186 29125 raft_consensus.cc:740] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.585248 29125 consensus_queue.cc:260] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.585251 29130 raft_consensus.cc:359] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.585310 29130 raft_consensus.cc:385] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.585326 29130 raft_consensus.cc:740] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.585368 29130 consensus_queue.cc:260] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.585446 29125 ts_tablet_manager.cc:1434] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:07:45.585453 29130 ts_tablet_manager.cc:1434] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.585527 29130 tablet_bootstrap.cc:492] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.585947 29141 raft_consensus.cc:493] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.585989 29130 tablet_bootstrap.cc:654] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.586000 29141 raft_consensus.cc:515] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.586130 29141 leader_election.cc:290] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.586596 29130 tablet_bootstrap.cc:492] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.586644 29130 ts_tablet_manager.cc:1403] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:45.586791 29130 raft_consensus.cc:359] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.586854 29130 raft_consensus.cc:385] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.586880 29130 raft_consensus.cc:740] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.586928 29130 consensus_queue.cc:260] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.587042 29130 ts_tablet_manager.cc:1434] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:07:45.587095 29130 tablet_bootstrap.cc:492] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec: Bootstrap starting.
I20251027 03:07:45.587466 29130 tablet_bootstrap.cc:654] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:45.589032 29130 tablet_bootstrap.cc:492] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec: No bootstrap required, opened a new log
I20251027 03:07:45.589077 29130 ts_tablet_manager.cc:1403] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:45.589144 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "211f82947754441f8f296df38ad1a0ba" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4" is_pre_election: true
I20251027 03:07:45.589217 29079 raft_consensus.cc:2468] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.589201 29130 raft_consensus.cc:359] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.589246 29130 raft_consensus.cc:385] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:45.589267 29130 raft_consensus.cc:740] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Initialized, Role: FOLLOWER
I20251027 03:07:45.589311 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "211f82947754441f8f296df38ad1a0ba" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.589323 29130 consensus_queue.cc:260] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.589388 29004 raft_consensus.cc:2468] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.589347 28892 leader_election.cc:304] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.589486 29141 raft_consensus.cc:2804] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.589494 29130 ts_tablet_manager.cc:1434] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:07:45.589530 29141 raft_consensus.cc:493] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.589553 29141 raft_consensus.cc:3060] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.590024 29141 raft_consensus.cc:515] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.590122 29141 leader_election.cc:290] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Requested vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.590256 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "211f82947754441f8f296df38ad1a0ba" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
I20251027 03:07:45.590253 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "211f82947754441f8f296df38ad1a0ba" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.590339 29079 raft_consensus.cc:3060] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.590351 29004 raft_consensus.cc:3060] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.590883 29079 raft_consensus.cc:2468] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.590929 29004 raft_consensus.cc:2468] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.591019 28892 leader_election.cc:304] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.591110 29141 raft_consensus.cc:2804] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.591218 29141 raft_consensus.cc:697] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Becoming Leader. State: Replica: 9652d571953345188c5ccaab47629bec, State: Running, Role: LEADER
I20251027 03:07:45.591261 29141 consensus_queue.cc:237] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.591697 28824 catalog_manager.cc:5649] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec reported cstate change: term changed from 0 to 1, leader changed from <none> to 9652d571953345188c5ccaab47629bec (127.27.151.65). New cstate: current_term: 1 leader_uuid: "9652d571953345188c5ccaab47629bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:45.597124 29128 raft_consensus.cc:493] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.597172 29128 raft_consensus.cc:515] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.597273 29128 leader_election.cc:290] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.597397 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90885115b0c240c89820cf90b0cf8bad" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec" is_pre_election: true
I20251027 03:07:45.597424 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90885115b0c240c89820cf90b0cf8bad" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.597447 28928 raft_consensus.cc:2468] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.597493 29004 raft_consensus.cc:2468] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.597596 29045 leader_election.cc:304] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 8e1ed5f3b1ac408091382b39763e63b4; no voters:
I20251027 03:07:45.597678 29128 raft_consensus.cc:2804] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.597718 29128 raft_consensus.cc:493] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.597733 29128 raft_consensus.cc:3060] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.598162 29128 raft_consensus.cc:515] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.598258 29128 leader_election.cc:290] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.598368 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90885115b0c240c89820cf90b0cf8bad" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec"
I20251027 03:07:45.598387 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90885115b0c240c89820cf90b0cf8bad" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.598414 28928 raft_consensus.cc:3060] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.598425 29004 raft_consensus.cc:3060] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.598811 29004 raft_consensus.cc:2468] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.598845 28928 raft_consensus.cc:2468] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.598942 29045 leader_election.cc:304] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.599015 29128 raft_consensus.cc:2804] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.599164 29128 raft_consensus.cc:697] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Running, Role: LEADER
I20251027 03:07:45.599205 29128 consensus_queue.cc:237] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.599606 28824 catalog_manager.cc:5649] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "8e1ed5f3b1ac408091382b39763e63b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:45.627228 29128 raft_consensus.cc:493] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.627285 29128 raft_consensus.cc:515] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.627388 29128 leader_election.cc:290] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 9652d571953345188c5ccaab47629bec (127.27.151.65:41377)
I20251027 03:07:45.627511 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cfff53f915f4279b49cc70c3fd67ee8" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec" is_pre_election: true
I20251027 03:07:45.627511 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cfff53f915f4279b49cc70c3fd67ee8" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.627579 28928 raft_consensus.cc:2468] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.627597 29004 raft_consensus.cc:2468] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.627678 29045 leader_election.cc:304] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.627753 29128 raft_consensus.cc:2804] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.627795 29128 raft_consensus.cc:493] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.627822 29128 raft_consensus.cc:3060] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.628320 29128 raft_consensus.cc:515] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.628410 29128 leader_election.cc:290] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Requested vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 9652d571953345188c5ccaab47629bec (127.27.151.65:41377)
I20251027 03:07:45.628551 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cfff53f915f4279b49cc70c3fd67ee8" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec"
I20251027 03:07:45.628604 28928 raft_consensus.cc:3060] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.628620 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2cfff53f915f4279b49cc70c3fd67ee8" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.628677 29004 raft_consensus.cc:3060] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.629117 29004 raft_consensus.cc:2468] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.629122 28928 raft_consensus.cc:2468] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.629274 29045 leader_election.cc:304] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.629369 29128 raft_consensus.cc:2804] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.629420 29128 raft_consensus.cc:697] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Running, Role: LEADER
I20251027 03:07:45.629462 29128 consensus_queue.cc:237] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } }
I20251027 03:07:45.629946 28824 catalog_manager.cc:5649] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "8e1ed5f3b1ac408091382b39763e63b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:45.633646 29128 raft_consensus.cc:493] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.633692 29128 raft_consensus.cc:515] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.633769 29128 leader_election.cc:290] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.633910 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.633913 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec" is_pre_election: true
I20251027 03:07:45.633956 29004 raft_consensus.cc:2468] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.633977 28928 raft_consensus.cc:2468] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.634061 29045 leader_election.cc:304] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 8e1ed5f3b1ac408091382b39763e63b4; no voters:
I20251027 03:07:45.634168 29128 raft_consensus.cc:2804] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.634202 29128 raft_consensus.cc:493] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.634214 29128 raft_consensus.cc:3060] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.634763 29128 raft_consensus.cc:515] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.634846 29128 leader_election.cc:290] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.634977 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec"
I20251027 03:07:45.635006 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.635036 28928 raft_consensus.cc:3060] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.635042 29004 raft_consensus.cc:3060] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.635479 28928 raft_consensus.cc:2468] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.635492 29004 raft_consensus.cc:2468] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.635643 29045 leader_election.cc:304] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.635725 29128 raft_consensus.cc:2804] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.635775 29128 raft_consensus.cc:697] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Running, Role: LEADER
I20251027 03:07:45.635831 29128 consensus_queue.cc:237] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.636302 28824 catalog_manager.cc:5649] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "8e1ed5f3b1ac408091382b39763e63b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:45.668324 29141 raft_consensus.cc:493] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.668380 29141 raft_consensus.cc:515] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.668471 29141 leader_election.cc:290] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.668627 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.668614 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4" is_pre_election: true
I20251027 03:07:45.668730 29004 raft_consensus.cc:2468] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.668744 29079 raft_consensus.cc:2468] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.668860 28894 leader_election.cc:304] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.668948 29141 raft_consensus.cc:2804] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.668983 29141 raft_consensus.cc:493] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.669001 29141 raft_consensus.cc:3060] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.669488 29141 raft_consensus.cc:515] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.669588 29141 leader_election.cc:290] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Requested vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.669726 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.669737 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
I20251027 03:07:45.669785 29004 raft_consensus.cc:3060] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.669799 29079 raft_consensus.cc:3060] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.670228 29004 raft_consensus.cc:2468] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.670264 29079 raft_consensus.cc:2468] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.670398 28894 leader_election.cc:304] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.670483 29141 raft_consensus.cc:2804] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.670527 29141 raft_consensus.cc:697] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Becoming Leader. State: Replica: 9652d571953345188c5ccaab47629bec, State: Running, Role: LEADER
I20251027 03:07:45.670578 29141 consensus_queue.cc:237] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.671042 28824 catalog_manager.cc:5649] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec reported cstate change: term changed from 0 to 1, leader changed from <none> to 9652d571953345188c5ccaab47629bec (127.27.151.65). New cstate: current_term: 1 leader_uuid: "9652d571953345188c5ccaab47629bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:45.694712 29141 raft_consensus.cc:493] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.694784 29141 raft_consensus.cc:515] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.694890 29141 leader_election.cc:290] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.695058 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89c9d44ada384f12a1365f54af14edb3" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.695124 29004 raft_consensus.cc:2468] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.695072 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89c9d44ada384f12a1365f54af14edb3" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4" is_pre_election: true
I20251027 03:07:45.695166 29079 raft_consensus.cc:2468] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.695258 28894 leader_election.cc:304] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.695363 29141 raft_consensus.cc:2804] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.695410 29141 raft_consensus.cc:493] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.695425 29141 raft_consensus.cc:3060] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.695919 29141 raft_consensus.cc:515] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.696022 29141 leader_election.cc:290] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Requested vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.696177 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89c9d44ada384f12a1365f54af14edb3" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.696173 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89c9d44ada384f12a1365f54af14edb3" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
I20251027 03:07:45.696247 29004 raft_consensus.cc:3060] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.696256 29079 raft_consensus.cc:3060] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.696719 29004 raft_consensus.cc:2468] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.696725 29079 raft_consensus.cc:2468] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.696889 28894 leader_election.cc:304] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.696977 29141 raft_consensus.cc:2804] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.697036 29141 raft_consensus.cc:697] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Becoming Leader. State: Replica: 9652d571953345188c5ccaab47629bec, State: Running, Role: LEADER
I20251027 03:07:45.697177 29141 consensus_queue.cc:237] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.697683 28824 catalog_manager.cc:5649] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec reported cstate change: term changed from 0 to 1, leader changed from <none> to 9652d571953345188c5ccaab47629bec (127.27.151.65). New cstate: current_term: 1 leader_uuid: "9652d571953345188c5ccaab47629bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: UNKNOWN } } }
W20251027 03:07:45.730878 28955 tablet.cc:2378] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 03:07:45.737578 29141 raft_consensus.cc:493] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.737639 29141 raft_consensus.cc:515] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.737767 29141 leader_election.cc:290] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.737907 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5876093fda44ea8bac93fdba6731a84" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.737910 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5876093fda44ea8bac93fdba6731a84" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4" is_pre_election: true
I20251027 03:07:45.737984 29004 raft_consensus.cc:2468] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.737988 29079 raft_consensus.cc:2468] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 0.
I20251027 03:07:45.738150 28894 leader_election.cc:304] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.738236 29141 raft_consensus.cc:2804] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.738273 29141 raft_consensus.cc:493] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.738293 29141 raft_consensus.cc:3060] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.738771 29141 raft_consensus.cc:515] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.738862 29141 leader_election.cc:290] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Requested vote from peers 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:45.739022 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5876093fda44ea8bac93fdba6731a84" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
I20251027 03:07:45.739022 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5876093fda44ea8bac93fdba6731a84" candidate_uuid: "9652d571953345188c5ccaab47629bec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.739120 29079 raft_consensus.cc:3060] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.739125 29004 raft_consensus.cc:3060] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.739681 29004 raft_consensus.cc:2468] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.739681 29079 raft_consensus.cc:2468] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9652d571953345188c5ccaab47629bec in term 1.
I20251027 03:07:45.739861 28894 leader_election.cc:304] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.739944 29141 raft_consensus.cc:2804] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.740039 29141 raft_consensus.cc:697] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Becoming Leader. State: Replica: 9652d571953345188c5ccaab47629bec, State: Running, Role: LEADER
I20251027 03:07:45.740095 29141 consensus_queue.cc:237] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.740672 28824 catalog_manager.cc:5649] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec reported cstate change: term changed from 0 to 1, leader changed from <none> to 9652d571953345188c5ccaab47629bec (127.27.151.65). New cstate: current_term: 1 leader_uuid: "9652d571953345188c5ccaab47629bec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:45.941196 29128 raft_consensus.cc:493] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:45.941303 29128 raft_consensus.cc:515] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.941442 29128 leader_election.cc:290] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.941694 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd52ac8fcb3a4558a20b88edc07b0ea1" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342" is_pre_election: true
I20251027 03:07:45.941756 29004 raft_consensus.cc:2468] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.941728 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd52ac8fcb3a4558a20b88edc07b0ea1" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec" is_pre_election: true
I20251027 03:07:45.941849 28928 raft_consensus.cc:2468] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 0.
I20251027 03:07:45.942013 29045 leader_election.cc:304] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e1ed5f3b1ac408091382b39763e63b4, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:45.942097 29128 raft_consensus.cc:2804] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:45.942135 29128 raft_consensus.cc:493] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:45.942157 29128 raft_consensus.cc:3060] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.942734 29128 raft_consensus.cc:515] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.942831 29128 leader_election.cc:290] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66:37695)
I20251027 03:07:45.942996 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd52ac8fcb3a4558a20b88edc07b0ea1" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9652d571953345188c5ccaab47629bec"
I20251027 03:07:45.943027 29004 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd52ac8fcb3a4558a20b88edc07b0ea1" candidate_uuid: "8e1ed5f3b1ac408091382b39763e63b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
I20251027 03:07:45.943086 28928 raft_consensus.cc:3060] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.943099 29004 raft_consensus.cc:3060] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:45.943545 29004 raft_consensus.cc:2468] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.943600 28928 raft_consensus.cc:2468] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e1ed5f3b1ac408091382b39763e63b4 in term 1.
I20251027 03:07:45.943702 29045 leader_election.cc:304] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 8e1ed5f3b1ac408091382b39763e63b4; no voters:
I20251027 03:07:45.943794 29128 raft_consensus.cc:2804] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:45.943849 29128 raft_consensus.cc:697] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Running, Role: LEADER
I20251027 03:07:45.943897 29128 consensus_queue.cc:237] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:45.944463 28824 catalog_manager.cc:5649] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "8e1ed5f3b1ac408091382b39763e63b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:45.992725 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:45.993906 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:45.994016 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:45.994082 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:45.994187 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:45.994282 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:45.994304 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:45.994315 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534465994316 us; error 0 us; skew 500 ppm
I20251027 03:07:45.995002 28253 webserver.cc:492] Webserver started at http://127.27.151.68:46609/ using document root <none> and password file <none>
I20251027 03:07:45.995091 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:45.995146 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:45.995211 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:45.995502 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "10a5a7756b66484191b93ed80c66b00c"
format_stamp: "Formatted at 2025-10-27 03:07:45 on dist-test-slave-4cc8"
I20251027 03:07:45.996709 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20251027 03:07:45.996958 29128 consensus_queue.cc:1048] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:45.997393 29179 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:45.997747 28253 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251027 03:07:45.997807 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "10a5a7756b66484191b93ed80c66b00c"
format_stamp: "Formatted at 2025-10-27 03:07:45 on dist-test-slave-4cc8"
I20251027 03:07:45.997865 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:45.998591 29162 consensus_queue.cc:1048] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.003257 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:46.003448 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:46.003558 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:46.003690 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:46.003711 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:46.003739 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:46.003762 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:46.006982 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:38137
I20251027 03:07:46.007007 29246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:38137 every 8 connection(s)
I20251027 03:07:46.008881 29160 consensus_queue.cc:1048] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.009905 29247 heartbeater.cc:344] Connected to a master server at 127.27.151.126:42585
I20251027 03:07:46.009956 29247 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:46.009961 29141 consensus_queue.cc:1048] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.010077 29247 heartbeater.cc:507] Master 127.27.151.126:42585 requested a full tablet report, sending...
I20251027 03:07:46.010458 28824 ts_manager.cc:194] Registered new tserver with Master: 10a5a7756b66484191b93ed80c66b00c (127.27.151.68:38137)
I20251027 03:07:46.010876 28824 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36652
I20251027 03:07:46.045760 29128 consensus_queue.cc:1048] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.046972 29162 consensus_queue.cc:1048] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.101649 29162 consensus_queue.cc:1048] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.102890 29255 consensus_queue.cc:1048] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.112835 29141 consensus_queue.cc:1048] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.113847 29160 consensus_queue.cc:1048] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.213901 29160 consensus_queue.cc:1048] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.215145 29260 consensus_queue.cc:1048] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.286549 29260 consensus_queue.cc:1048] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.288003 29160 consensus_queue.cc:1048] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.423380 29255 consensus_queue.cc:1048] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.424736 29128 consensus_queue.cc:1048] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:46.431665 28880 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:07:46.434280 28928 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa"
dest_uuid: "9652d571953345188c5ccaab47629bec"
mode: GRACEFUL
new_leader_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
from {username='slave'} at 127.0.0.1:33092
I20251027 03:07:46.434352 28928 raft_consensus.cc:606] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Received request to transfer leadership to TS 182815b12fa94a2b88c3ddc0b49e4342
I20251027 03:07:46.436642 29079 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612"
dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
mode: GRACEFUL
new_leader_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
from {username='slave'} at 127.0.0.1:36004
I20251027 03:07:46.436697 29079 raft_consensus.cc:606] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Received request to transfer leadership to TS 182815b12fa94a2b88c3ddc0b49e4342
I20251027 03:07:46.436914 28880 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251027 03:07:46.436980 28880 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:07:46.588404 29128 raft_consensus.cc:993] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: : Instructing follower 182815b12fa94a2b88c3ddc0b49e4342 to start an election
I20251027 03:07:46.588481 29128 raft_consensus.cc:1081] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Signalling peer 182815b12fa94a2b88c3ddc0b49e4342 to start an election
I20251027 03:07:46.588740 29004 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612"
dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
from {username='slave'} at 127.0.0.1:34360
I20251027 03:07:46.588819 29004 raft_consensus.cc:493] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:07:46.588840 29004 raft_consensus.cc:3060] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:07:46.589392 29004 raft_consensus.cc:515] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:46.589550 29004 leader_election.cc:290] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [CANDIDATE]: Term 2 election: Requested vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:46.591902 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612" candidate_uuid: "182815b12fa94a2b88c3ddc0b49e4342" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
I20251027 03:07:46.591966 29079 raft_consensus.cc:3055] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:07:46.591991 29079 raft_consensus.cc:740] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8e1ed5f3b1ac408091382b39763e63b4, State: Running, Role: LEADER
I20251027 03:07:46.592017 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59285374574c4ce6b2a2612a90e60612" candidate_uuid: "182815b12fa94a2b88c3ddc0b49e4342" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9652d571953345188c5ccaab47629bec"
I20251027 03:07:46.592051 29079 consensus_queue.cc:260] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:46.592113 28928 raft_consensus.cc:3060] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:07:46.592182 29079 raft_consensus.cc:3060] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:07:46.592669 28928 raft_consensus.cc:2468] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182815b12fa94a2b88c3ddc0b49e4342 in term 2.
I20251027 03:07:46.592671 29079 raft_consensus.cc:2468] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182815b12fa94a2b88c3ddc0b49e4342 in term 2.
I20251027 03:07:46.592885 28970 leader_election.cc:304] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:46.593005 29142 raft_consensus.cc:2804] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:07:46.593111 29142 raft_consensus.cc:697] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 LEADER]: Becoming Leader. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Running, Role: LEADER
I20251027 03:07:46.593165 29142 consensus_queue.cc:237] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:46.593555 28823 catalog_manager.cc:5649] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 reported cstate change: term changed from 1 to 2, leader changed from 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67) to 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66). New cstate: current_term: 2 leader_uuid: "182815b12fa94a2b88c3ddc0b49e4342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:46.597460 29260 raft_consensus.cc:993] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: : Instructing follower 182815b12fa94a2b88c3ddc0b49e4342 to start an election
I20251027 03:07:46.597505 29260 raft_consensus.cc:1081] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Signalling peer 182815b12fa94a2b88c3ddc0b49e4342 to start an election
I20251027 03:07:46.597636 29004 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa"
dest_uuid: "182815b12fa94a2b88c3ddc0b49e4342"
from {username='slave'} at 127.0.0.1:34362
I20251027 03:07:46.597687 29004 raft_consensus.cc:493] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:07:46.597705 29004 raft_consensus.cc:3060] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:07:46.598116 29004 raft_consensus.cc:515] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:46.598208 29004 leader_election.cc:290] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [CANDIDATE]: Term 2 election: Requested vote from peers 9652d571953345188c5ccaab47629bec (127.27.151.65:41377), 8e1ed5f3b1ac408091382b39763e63b4 (127.27.151.67:35899)
I20251027 03:07:46.598323 28928 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa" candidate_uuid: "182815b12fa94a2b88c3ddc0b49e4342" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9652d571953345188c5ccaab47629bec"
I20251027 03:07:46.598387 28928 raft_consensus.cc:3055] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:07:46.598421 28928 raft_consensus.cc:740] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9652d571953345188c5ccaab47629bec, State: Running, Role: LEADER
I20251027 03:07:46.598474 28928 consensus_queue.cc:260] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:46.598554 28928 raft_consensus.cc:3060] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:07:46.598977 28928 raft_consensus.cc:2468] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182815b12fa94a2b88c3ddc0b49e4342 in term 2.
I20251027 03:07:46.599059 29079 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4e2b554f96a148edb80ce6f008a46baa" candidate_uuid: "182815b12fa94a2b88c3ddc0b49e4342" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8e1ed5f3b1ac408091382b39763e63b4"
I20251027 03:07:46.599107 29079 raft_consensus.cc:3060] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:07:46.599133 28970 leader_election.cc:304] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182815b12fa94a2b88c3ddc0b49e4342, 9652d571953345188c5ccaab47629bec; no voters:
I20251027 03:07:46.599257 29142 raft_consensus.cc:2804] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:07:46.599313 29142 raft_consensus.cc:697] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 LEADER]: Becoming Leader. State: Replica: 182815b12fa94a2b88c3ddc0b49e4342, State: Running, Role: LEADER
I20251027 03:07:46.599359 29142 consensus_queue.cc:237] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } }
I20251027 03:07:46.599642 29079 raft_consensus.cc:2468] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182815b12fa94a2b88c3ddc0b49e4342 in term 2.
I20251027 03:07:46.599746 28823 catalog_manager.cc:5649] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 reported cstate change: term changed from 1 to 2, leader changed from 9652d571953345188c5ccaab47629bec (127.27.151.65) to 182815b12fa94a2b88c3ddc0b49e4342 (127.27.151.66). New cstate: current_term: 2 leader_uuid: "182815b12fa94a2b88c3ddc0b49e4342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:47.011320 29247 heartbeater.cc:499] Master 127.27.151.126:42585 was elected leader, sending a full tablet report...
I20251027 03:07:47.059219 28928 raft_consensus.cc:1275] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:07:47.059432 29142 consensus_queue.cc:1048] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:47.060485 29079 raft_consensus.cc:1275] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:07:47.060635 29142 consensus_queue.cc:1048] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:47.067052 29079 raft_consensus.cc:1275] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:07:47.067238 29271 consensus_queue.cc:1048] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:47.068086 28928 raft_consensus.cc:1275] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:07:47.068331 29142 consensus_queue.cc:1048] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:48.437160 28880 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:07:48.437354 28880 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:07:48.437398 28880 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:07:50.437618 28880 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:07:50.437834 28880 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:07:50.437875 28880 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:07:51.437892 28928 consensus_queue.cc:237] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } }
I20251027 03:07:51.438462 29004 raft_consensus.cc:1275] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Refusing update from remote peer 9652d571953345188c5ccaab47629bec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:51.438593 29079 raft_consensus.cc:1275] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Refusing update from remote peer 9652d571953345188c5ccaab47629bec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:51.438659 29286 consensus_queue.cc:1048] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:51.438824 29285 consensus_queue.cc:1048] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:51.439486 29285 raft_consensus.cc:2955] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.439599 29004 raft_consensus.cc:2955] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.440661 29079 raft_consensus.cc:2955] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.440689 28823 catalog_manager.cc:5649] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 reported cstate change: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "9652d571953345188c5ccaab47629bec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } attrs { replace: true } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
W20251027 03:07:51.442090 28892 consensus_peers.cc:597] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec -> Peer 10a5a7756b66484191b93ed80c66b00c (127.27.151.68:38137): Couldn't send request to peer 10a5a7756b66484191b93ed80c66b00c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5876093fda44ea8bac93fdba6731a84. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:07:51.443920 29004 consensus_queue.cc:237] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } }
I20251027 03:07:51.444377 28928 raft_consensus.cc:1275] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:07:51.444491 29079 raft_consensus.cc:1275] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:07:51.444584 29280 consensus_queue.cc:1048] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:07:51.444674 29297 consensus_queue.cc:1048] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:07:51.445305 29297 raft_consensus.cc:2955] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.445571 28928 raft_consensus.cc:2955] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.446336 28825 catalog_manager.cc:5649] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 reported cstate change: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New cstate: current_term: 2 leader_uuid: "182815b12fa94a2b88c3ddc0b49e4342" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:51.446543 29004 consensus_queue.cc:237] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } }
I20251027 03:07:51.446362 29079 raft_consensus.cc:2955] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.447237 29078 raft_consensus.cc:1275] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20251027 03:07:51.447654 28968 consensus_peers.cc:597] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 -> Peer 10a5a7756b66484191b93ed80c66b00c (127.27.151.68:38137): Couldn't send request to peer 10a5a7756b66484191b93ed80c66b00c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2b554f96a148edb80ce6f008a46baa. This is attempt 1: this message will repeat every 5th retry.
W20251027 03:07:51.447703 28968 consensus_peers.cc:597] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 -> Peer 10a5a7756b66484191b93ed80c66b00c (127.27.151.68:38137): Couldn't send request to peer 10a5a7756b66484191b93ed80c66b00c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59285374574c4ce6b2a2612a90e60612. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:07:51.447897 29303 consensus_queue.cc:1048] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:07:51.447974 28928 raft_consensus.cc:1275] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Refusing update from remote peer 182815b12fa94a2b88c3ddc0b49e4342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:07:51.448199 29303 consensus_queue.cc:1048] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:07:51.448566 29301 raft_consensus.cc:2955] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.448661 29079 raft_consensus.cc:2955] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.448915 28928 raft_consensus.cc:2955] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.449543 28824 catalog_manager.cc:5649] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 reported cstate change: config changed from index -1 to 3, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New cstate: current_term: 2 leader_uuid: "182815b12fa94a2b88c3ddc0b49e4342" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } attrs { replace: true } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.449879 28928 consensus_queue.cc:237] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } }
I20251027 03:07:51.450280 29004 raft_consensus.cc:1275] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Refusing update from remote peer 9652d571953345188c5ccaab47629bec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251027 03:07:51.450428 28892 consensus_peers.cc:597] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec -> Peer 10a5a7756b66484191b93ed80c66b00c (127.27.151.68:38137): Couldn't send request to peer 10a5a7756b66484191b93ed80c66b00c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89c9d44ada384f12a1365f54af14edb3. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:07:51.450501 29079 raft_consensus.cc:1275] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Refusing update from remote peer 9652d571953345188c5ccaab47629bec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:51.450491 29285 consensus_queue.cc:1048] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:51.450659 29285 consensus_queue.cc:1048] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:07:51.451160 29281 raft_consensus.cc:2955] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.451251 29004 raft_consensus.cc:2955] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.451938 29079 raft_consensus.cc:2955] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } } }
I20251027 03:07:51.451990 28825 catalog_manager.cc:5649] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec reported cstate change: config changed from index -1 to 2, NON_VOTER 10a5a7756b66484191b93ed80c66b00c (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "9652d571953345188c5ccaab47629bec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9652d571953345188c5ccaab47629bec" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "182815b12fa94a2b88c3ddc0b49e4342" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 37695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1ed5f3b1ac408091382b39763e63b4" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10a5a7756b66484191b93ed80c66b00c" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 38137 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:51.520967 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
W20251027 03:07:51.521795 28812 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20251027 03:07:51.522204 28874 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251027 03:07:51.522274 28874 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251027 03:07:51.522722 28874 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251027 03:07:51.522845 28874 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251027 03:07:51.524452 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:51.524538 28253 tablet_replica.cc:333] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.524591 28253 raft_consensus.cc:2243] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.524665 28253 raft_consensus.cc:2272] T a5876093fda44ea8bac93fdba6731a84 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.524922 28253 tablet_replica.cc:333] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.524976 28253 raft_consensus.cc:2243] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.525007 28253 raft_consensus.cc:2272] T 4e2b554f96a148edb80ce6f008a46baa P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.525177 28253 tablet_replica.cc:333] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.525233 28253 raft_consensus.cc:2243] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.525267 28253 raft_consensus.cc:2272] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.525377 28253 tablet_replica.cc:333] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.525416 28253 raft_consensus.cc:2243] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.525475 28253 raft_consensus.cc:2272] T 89c9d44ada384f12a1365f54af14edb3 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.525676 28253 tablet_replica.cc:333] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.525723 28253 raft_consensus.cc:2243] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.525758 28253 raft_consensus.cc:2272] T 90885115b0c240c89820cf90b0cf8bad P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.525878 28253 tablet_replica.cc:333] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.525921 28253 raft_consensus.cc:2243] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.525957 28253 raft_consensus.cc:2272] T 59285374574c4ce6b2a2612a90e60612 P 9652d571953345188c5ccaab47629bec [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.526139 28253 tablet_replica.cc:333] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.526199 28253 raft_consensus.cc:2243] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.526252 28253 raft_consensus.cc:2272] T 211f82947754441f8f296df38ad1a0ba P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.526364 28253 tablet_replica.cc:333] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec: stopping tablet replica
I20251027 03:07:51.526402 28253 raft_consensus.cc:2243] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.526438 28253 raft_consensus.cc:2272] T 2cfff53f915f4279b49cc70c3fd67ee8 P 9652d571953345188c5ccaab47629bec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.528610 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:07:51.529809 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
W20251027 03:07:51.530404 28874 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251027 03:07:51.530462 28874 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251027 03:07:51.531128 28874 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251027 03:07:51.531222 28874 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251027 03:07:51.533500 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:51.533586 28253 tablet_replica.cc:333] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.533643 28253 raft_consensus.cc:2243] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.533701 28253 raft_consensus.cc:2272] T 4e2b554f96a148edb80ce6f008a46baa P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.533840 28253 tablet_replica.cc:333] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.533874 28253 raft_consensus.cc:2243] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.533891 28253 raft_consensus.cc:2272] T 2cfff53f915f4279b49cc70c3fd67ee8 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.533962 28253 tablet_replica.cc:333] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.533986 28253 raft_consensus.cc:2243] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.534017 28253 raft_consensus.cc:2272] T 59285374574c4ce6b2a2612a90e60612 P 182815b12fa94a2b88c3ddc0b49e4342 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.534217 28253 tablet_replica.cc:333] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.534255 28253 raft_consensus.cc:2243] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.534279 28253 raft_consensus.cc:2272] T 211f82947754441f8f296df38ad1a0ba P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.534358 28253 tablet_replica.cc:333] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.534382 28253 raft_consensus.cc:2243] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.534402 28253 raft_consensus.cc:2272] T 90885115b0c240c89820cf90b0cf8bad P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.534467 28253 tablet_replica.cc:333] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.534493 28253 raft_consensus.cc:2243] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.534512 28253 raft_consensus.cc:2272] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.534576 28253 tablet_replica.cc:333] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.534601 28253 raft_consensus.cc:2243] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.534623 28253 raft_consensus.cc:2272] T 89c9d44ada384f12a1365f54af14edb3 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.534783 28253 tablet_replica.cc:333] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342: stopping tablet replica
I20251027 03:07:51.534821 28253 raft_consensus.cc:2243] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.534847 28253 raft_consensus.cc:2272] T a5876093fda44ea8bac93fdba6731a84 P 182815b12fa94a2b88c3ddc0b49e4342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.536760 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:07:51.537667 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:07:51.540930 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:51.541021 28253 tablet_replica.cc:333] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.541081 28253 raft_consensus.cc:2243] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.541127 28253 raft_consensus.cc:2272] T 4e2b554f96a148edb80ce6f008a46baa P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.541303 28253 tablet_replica.cc:333] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.541339 28253 raft_consensus.cc:2243] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.541384 28253 raft_consensus.cc:2272] T 2cfff53f915f4279b49cc70c3fd67ee8 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.541489 28253 tablet_replica.cc:333] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.541518 28253 raft_consensus.cc:2243] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.541554 28253 raft_consensus.cc:2272] T a5876093fda44ea8bac93fdba6731a84 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.541689 28253 tablet_replica.cc:333] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.541720 28253 raft_consensus.cc:2243] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.541769 28253 raft_consensus.cc:2272] T dd52ac8fcb3a4558a20b88edc07b0ea1 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.541867 28253 tablet_replica.cc:333] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.541898 28253 raft_consensus.cc:2243] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.541924 28253 raft_consensus.cc:2272] T 89c9d44ada384f12a1365f54af14edb3 P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.542095 28253 tablet_replica.cc:333] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.542128 28253 raft_consensus.cc:2243] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.542151 28253 raft_consensus.cc:2272] T 211f82947754441f8f296df38ad1a0ba P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.542243 28253 tablet_replica.cc:333] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.542271 28253 raft_consensus.cc:2243] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.542327 28253 raft_consensus.cc:2272] T 90885115b0c240c89820cf90b0cf8bad P 8e1ed5f3b1ac408091382b39763e63b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.542415 28253 tablet_replica.cc:333] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4: stopping tablet replica
I20251027 03:07:51.542443 28253 raft_consensus.cc:2243] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:51.542479 28253 raft_consensus.cc:2272] T 59285374574c4ce6b2a2612a90e60612 P 8e1ed5f3b1ac408091382b39763e63b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.553895 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:07:51.554885 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:07:51.557428 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:51.568809 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:07:51.569380 28253 master.cc:561] Master@127.27.151.126:42585 shutting down...
I20251027 03:07:51.571877 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:51.571925 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:51.571940 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P ef229b83cee04555b0e559ac37fb88a0: stopping tablet replica
I20251027 03:07:51.573962 28253 master.cc:583] Master@127.27.151.126:42585 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7172 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20251027 03:07:51.581347 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:46131,127.27.151.125:45191,127.27.151.124:39135
I20251027 03:07:51.581466 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:51.582885 29311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.582907 29313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.583031 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:51.583024 29310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.583215 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:51.583248 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:51.583271 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534471583271 us; error 0 us; skew 500 ppm
I20251027 03:07:51.583773 28253 webserver.cc:492] Webserver started at http://127.27.151.126:39459/ using document root <none> and password file <none>
I20251027 03:07:51.583851 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:51.583897 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:51.583956 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:51.584239 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "03699e34e04f4250aa533c9b43777b86"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.585117 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:51.585492 29318 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.585625 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.585675 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "03699e34e04f4250aa533c9b43777b86"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.585737 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:51.591336 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:51.591470 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:51.593736 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:46131
I20251027 03:07:51.594270 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:07:51.598649 29370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:46131 every 8 connection(s)
I20251027 03:07:51.598752 29371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251027 03:07:51.599357 29373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.599354 29376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.599797 29371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:51.600605 29374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.600836 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:51.600939 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:51.600962 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:51.600982 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534471600982 us; error 0 us; skew 500 ppm
I20251027 03:07:51.601503 28253 webserver.cc:492] Webserver started at http://127.27.151.125:35339/ using document root <none> and password file <none>
I20251027 03:07:51.601579 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:51.601622 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:51.601668 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:51.601897 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root/instance:
uuid: "0815b60b50e44901ad5ec34e006374d7"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.602886 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:51.603341 29383 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.603484 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.603525 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root
uuid: "0815b60b50e44901ad5ec34e006374d7"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.603567 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:51.604923 29371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:51.605850 29371 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.151.125:45191: Network error: Client connection negotiation failed: client connection to 127.27.151.125:45191: connect: Connection refused (error 111)
I20251027 03:07:51.609845 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:51.610003 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:51.612264 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.125:45191
I20251027 03:07:51.612290 29436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.125:45191 every 8 connection(s)
I20251027 03:07:51.612510 29437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:51.612568 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:07:51.613251 29437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:51.613518 29439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.613528 29442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.613673 29440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.613945 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:51.614053 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:51.614081 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:51.614105 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534471614105 us; error 0 us; skew 500 ppm
I20251027 03:07:51.614578 28253 webserver.cc:492] Webserver started at http://127.27.151.124:36449/ using document root <none> and password file <none>
I20251027 03:07:51.614651 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:51.614712 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:51.614760 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:51.614980 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root/instance:
uuid: "c5ac390d1adc47d48e4c54812ba085d3"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.615921 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:51.616271 29437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:51.616585 29448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.616694 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.616750 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root
uuid: "c5ac390d1adc47d48e4c54812ba085d3"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.616801 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:51.619254 29437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } has no permanent_uuid. Determining permanent_uuid...
W20251027 03:07:51.619725 29437 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.27.151.124:39135: Network error: Client connection negotiation failed: client connection to 127.27.151.124:39135: connect: Connection refused (error 111)
I20251027 03:07:51.631356 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:51.631521 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:51.634104 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.124:39135
I20251027 03:07:51.634130 29502 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.124:39135 every 8 connection(s)
I20251027 03:07:51.634311 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:07:51.634392 29503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:51.634922 29503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:51.637259 29503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:51.639554 29503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:51.642469 29503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: Bootstrap starting.
I20251027 03:07:51.642799 29503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:51.643282 29503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: No bootstrap required, opened a new log
I20251027 03:07:51.643420 29503 raft_consensus.cc:359] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.643468 29503 raft_consensus.cc:385] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:51.643488 29503 raft_consensus.cc:740] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ac390d1adc47d48e4c54812ba085d3, State: Initialized, Role: FOLLOWER
I20251027 03:07:51.643538 29503 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.643688 29503 sys_catalog.cc:565] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:51.643674 29508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.643746 29508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: This master's current role is: FOLLOWER
W20251027 03:07:51.644763 29519 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 03:07:51.644804 29519 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 03:07:51.646876 29437 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } attempt: 1
I20251027 03:07:51.649405 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: Bootstrap starting.
I20251027 03:07:51.649770 29437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:51.650259 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: No bootstrap required, opened a new log
I20251027 03:07:51.650384 29437 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.650429 29437 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:51.650441 29437 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0815b60b50e44901ad5ec34e006374d7, State: Initialized, Role: FOLLOWER
I20251027 03:07:51.650487 29437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.650638 29437 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:51.650650 29521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.650699 29521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: This master's current role is: FOLLOWER
W20251027 03:07:51.651641 29532 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 03:07:51.651674 29532 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 03:07:51.652985 29371 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } attempt: 1
I20251027 03:07:51.655133 29371 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } has no permanent_uuid. Determining permanent_uuid...
I20251027 03:07:51.657657 29371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86: Bootstrap starting.
I20251027 03:07:51.658022 29371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:51.658485 29371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86: No bootstrap required, opened a new log
I20251027 03:07:51.658605 29371 raft_consensus.cc:359] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.658651 29371 raft_consensus.cc:385] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:51.658664 29371 raft_consensus.cc:740] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03699e34e04f4250aa533c9b43777b86, State: Initialized, Role: FOLLOWER
I20251027 03:07:51.658711 29371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.658857 29371 sys_catalog.cc:565] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:51.658871 29534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.658924 29534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:51.659881 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20251027 03:07:51.659922 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20251027 03:07:51.659974 29545 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 03:07:51.660010 29545 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 03:07:51.660097 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:51.661146 29546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.661176 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:51.661298 29547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.661307 29549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.661507 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:51.661542 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:51.661556 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534471661556 us; error 0 us; skew 500 ppm
I20251027 03:07:51.662041 28253 webserver.cc:492] Webserver started at http://127.27.151.65:43899/ using document root <none> and password file <none>
I20251027 03:07:51.662115 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:51.662159 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:51.662207 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:51.662456 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "a3f6f2aeeceb4b46a192d86970b401d6"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.663223 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:51.663589 29554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.663691 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.663733 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "a3f6f2aeeceb4b46a192d86970b401d6"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.663772 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:51.713490 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:51.713738 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:51.713940 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:51.714161 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:51.714190 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.714221 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:51.714249 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.717393 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:46841
I20251027 03:07:51.717459 29617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:46841 every 8 connection(s)
I20251027 03:07:51.717868 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:51.719457 29626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.720139 29628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.720379 29625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.720909 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:51.720923 29620 heartbeater.cc:344] Connected to a master server at 127.27.151.125:45191
I20251027 03:07:51.721014 29620 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.721050 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:51.721079 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:51.721096 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534471721096 us; error 0 us; skew 500 ppm
I20251027 03:07:51.721138 29620 heartbeater.cc:507] Master 127.27.151.125:45191 requested a full tablet report, sending...
I20251027 03:07:51.721484 29401 ts_manager.cc:194] Registered new tserver with Master: a3f6f2aeeceb4b46a192d86970b401d6 (127.27.151.65:46841)
I20251027 03:07:51.721760 29618 heartbeater.cc:344] Connected to a master server at 127.27.151.124:39135
I20251027 03:07:51.721844 28253 webserver.cc:492] Webserver started at http://127.27.151.66:43977/ using document root <none> and password file <none>
I20251027 03:07:51.721918 29618 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.721962 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:51.722013 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:51.722023 29618 heartbeater.cc:507] Master 127.27.151.124:39135 requested a full tablet report, sending...
I20251027 03:07:51.722060 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:51.722321 29467 ts_manager.cc:194] Registered new tserver with Master: a3f6f2aeeceb4b46a192d86970b401d6 (127.27.151.65:46841)
I20251027 03:07:51.722376 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "71abed830bde4ebda19c38634a581a90"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.722568 29619 heartbeater.cc:344] Connected to a master server at 127.27.151.126:46131
I20251027 03:07:51.722620 29619 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.722746 29619 heartbeater.cc:507] Master 127.27.151.126:46131 requested a full tablet report, sending...
I20251027 03:07:51.723019 29335 ts_manager.cc:194] Registered new tserver with Master: a3f6f2aeeceb4b46a192d86970b401d6 (127.27.151.65:46841)
I20251027 03:07:51.723421 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:51.723753 29633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.723853 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.723884 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "71abed830bde4ebda19c38634a581a90"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.723907 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:51.737215 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:51.737385 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:51.737494 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:51.737623 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:51.737644 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.737674 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:51.737686 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.740685 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:33773
I20251027 03:07:51.740742 29696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:33773 every 8 connection(s)
I20251027 03:07:51.741158 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:51.742456 29704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.742780 29703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:51.742934 29706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:51.744364 29699 heartbeater.cc:344] Connected to a master server at 127.27.151.125:45191
I20251027 03:07:51.744418 29699 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.744529 29699 heartbeater.cc:507] Master 127.27.151.125:45191 requested a full tablet report, sending...
I20251027 03:07:51.745057 29401 ts_manager.cc:194] Registered new tserver with Master: 71abed830bde4ebda19c38634a581a90 (127.27.151.66:33773)
I20251027 03:07:51.745345 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:51.745509 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:51.745543 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:51.745559 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534471745560 us; error 0 us; skew 500 ppm
I20251027 03:07:51.746163 28253 webserver.cc:492] Webserver started at http://127.27.151.67:36939/ using document root <none> and password file <none>
I20251027 03:07:51.746248 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:51.746289 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:51.746325 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:51.746575 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "7c6aacdcded44c93b5af1fd084a75c8d"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.747370 29697 heartbeater.cc:344] Connected to a master server at 127.27.151.124:39135
I20251027 03:07:51.747462 29697 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.747619 29697 heartbeater.cc:507] Master 127.27.151.124:39135 requested a full tablet report, sending...
I20251027 03:07:51.748090 29467 ts_manager.cc:194] Registered new tserver with Master: 71abed830bde4ebda19c38634a581a90 (127.27.151.66:33773)
I20251027 03:07:51.748246 29698 heartbeater.cc:344] Connected to a master server at 127.27.151.126:46131
I20251027 03:07:51.748287 29698 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.748373 29698 heartbeater.cc:507] Master 127.27.151.126:46131 requested a full tablet report, sending...
I20251027 03:07:51.748505 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20251027 03:07:51.748615 29335 ts_manager.cc:194] Registered new tserver with Master: 71abed830bde4ebda19c38634a581a90 (127.27.151.66:33773)
I20251027 03:07:51.749066 29711 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.749689 28253 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20251027 03:07:51.749747 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "7c6aacdcded44c93b5af1fd084a75c8d"
format_stamp: "Formatted at 2025-10-27 03:07:51 on dist-test-slave-4cc8"
I20251027 03:07:51.749794 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:51.753615 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:51.753832 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:51.753958 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:51.754123 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:51.754151 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.754179 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:51.754206 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:51.757218 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:35985
I20251027 03:07:51.757270 29774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:35985 every 8 connection(s)
I20251027 03:07:51.760388 29775 heartbeater.cc:344] Connected to a master server at 127.27.151.124:39135
I20251027 03:07:51.760445 29775 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.760495 29776 heartbeater.cc:344] Connected to a master server at 127.27.151.126:46131
I20251027 03:07:51.760530 29776 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.760540 29777 heartbeater.cc:344] Connected to a master server at 127.27.151.125:45191
I20251027 03:07:51.760555 29775 heartbeater.cc:507] Master 127.27.151.124:39135 requested a full tablet report, sending...
I20251027 03:07:51.760596 29777 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:51.760596 29776 heartbeater.cc:507] Master 127.27.151.126:46131 requested a full tablet report, sending...
I20251027 03:07:51.760689 29777 heartbeater.cc:507] Master 127.27.151.125:45191 requested a full tablet report, sending...
I20251027 03:07:51.760816 29467 ts_manager.cc:194] Registered new tserver with Master: 7c6aacdcded44c93b5af1fd084a75c8d (127.27.151.67:35985)
I20251027 03:07:51.760854 29335 ts_manager.cc:194] Registered new tserver with Master: 7c6aacdcded44c93b5af1fd084a75c8d (127.27.151.67:35985)
I20251027 03:07:51.761061 29401 ts_manager.cc:194] Registered new tserver with Master: 7c6aacdcded44c93b5af1fd084a75c8d (127.27.151.67:35985)
I20251027 03:07:51.761864 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004402181s
I20251027 03:07:51.928531 29534 raft_consensus.cc:493] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:51.928635 29534 raft_consensus.cc:515] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.928802 29534 leader_election.cc:290] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0815b60b50e44901ad5ec34e006374d7 (127.27.151.125:45191), c5ac390d1adc47d48e4c54812ba085d3 (127.27.151.124:39135)
I20251027 03:07:51.929039 29477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03699e34e04f4250aa533c9b43777b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ac390d1adc47d48e4c54812ba085d3" is_pre_election: true
I20251027 03:07:51.929037 29411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03699e34e04f4250aa533c9b43777b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0815b60b50e44901ad5ec34e006374d7" is_pre_election: true
I20251027 03:07:51.929158 29477 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03699e34e04f4250aa533c9b43777b86 in term 0.
I20251027 03:07:51.929164 29411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03699e34e04f4250aa533c9b43777b86 in term 0.
I20251027 03:07:51.929302 29321 leader_election.cc:304] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03699e34e04f4250aa533c9b43777b86, 0815b60b50e44901ad5ec34e006374d7; no voters:
I20251027 03:07:51.929385 29534 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:51.929421 29534 raft_consensus.cc:493] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:51.929445 29534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:51.930075 29534 raft_consensus.cc:515] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.930189 29534 leader_election.cc:290] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [CANDIDATE]: Term 1 election: Requested vote from peers 0815b60b50e44901ad5ec34e006374d7 (127.27.151.125:45191), c5ac390d1adc47d48e4c54812ba085d3 (127.27.151.124:39135)
I20251027 03:07:51.930335 29411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03699e34e04f4250aa533c9b43777b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0815b60b50e44901ad5ec34e006374d7"
I20251027 03:07:51.930335 29477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03699e34e04f4250aa533c9b43777b86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ac390d1adc47d48e4c54812ba085d3"
I20251027 03:07:51.930444 29411 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:51.930455 29477 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:51.931188 29411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03699e34e04f4250aa533c9b43777b86 in term 1.
I20251027 03:07:51.931187 29477 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03699e34e04f4250aa533c9b43777b86 in term 1.
I20251027 03:07:51.931391 29322 leader_election.cc:304] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03699e34e04f4250aa533c9b43777b86, c5ac390d1adc47d48e4c54812ba085d3; no voters:
I20251027 03:07:51.931473 29534 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:51.931597 29534 raft_consensus.cc:697] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 1 LEADER]: Becoming Leader. State: Replica: 03699e34e04f4250aa533c9b43777b86, State: Running, Role: LEADER
I20251027 03:07:51.931641 29534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:51.931915 29783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03699e34e04f4250aa533c9b43777b86. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.931979 29783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:51.932121 29785 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:07:51.932281 29785 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:07:51.932821 29477 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Refusing update from remote peer 03699e34e04f4250aa533c9b43777b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:51.932924 29411 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Refusing update from remote peer 03699e34e04f4250aa533c9b43777b86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:07:51.933017 29534 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:51.933075 29783 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:51.933604 29508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03699e34e04f4250aa533c9b43777b86. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.933638 29521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03699e34e04f4250aa533c9b43777b86. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.933684 29508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:51.933704 29521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:51.934041 29783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.934099 29783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:51.934158 29785 catalog_manager.cc:1357] Generated new cluster ID: 2a11fecd9bb6485eb76a1fe3902a9041
I20251027 03:07:51.934204 29785 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:07:51.934307 29508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.934366 29508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:51.934346 29783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.934413 29783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:51.934674 29521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03699e34e04f4250aa533c9b43777b86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:51.934738 29521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:51.941416 29785 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:07:51.941880 29785 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:07:51.948217 29785 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86: Generated new TSK 0
I20251027 03:07:51.948292 29785 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:07:52.645073 29519 catalog_manager.cc:1269] Loaded cluster ID: 2a11fecd9bb6485eb76a1fe3902a9041
I20251027 03:07:52.645164 29519 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: loading cluster ID for follower catalog manager: success
I20251027 03:07:52.645849 29519 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: acquiring CA information for follower catalog manager: success
I20251027 03:07:52.646023 29519 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 03:07:52.651862 29532 catalog_manager.cc:1269] Loaded cluster ID: 2a11fecd9bb6485eb76a1fe3902a9041
I20251027 03:07:52.651914 29532 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: loading cluster ID for follower catalog manager: success
I20251027 03:07:52.652530 29532 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: acquiring CA information for follower catalog manager: success
I20251027 03:07:52.652619 29532 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 03:07:52.723927 29335 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45842
I20251027 03:07:52.749375 29335 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45852
I20251027 03:07:52.761673 29335 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45858
I20251027 03:07:53.659922 29544 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:07:53.724426 29619 heartbeater.cc:499] Master 127.27.151.126:46131 was elected leader, sending a full tablet report...
I20251027 03:07:53.749817 29698 heartbeater.cc:499] Master 127.27.151.126:46131 was elected leader, sending a full tablet report...
I20251027 03:07:53.762130 29776 heartbeater.cc:499] Master 127.27.151.126:46131 was elected leader, sending a full tablet report...
I20251027 03:07:54.011473 28253 test_util.cc:276] Using random seed: 882197325
I20251027 03:07:54.016963 29335 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45870:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 03:07:54.022782 29582 tablet_service.cc:1505] Processing CreateTablet for tablet 019ac74f34d64573ae4ae976ee2d5bb9 (DEFAULT_TABLE table=test-workload [id=5b7ebd0edd6c4642a2828858c9701b86]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251027 03:07:54.022938 29582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019ac74f34d64573ae4ae976ee2d5bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:54.023586 29739 tablet_service.cc:1505] Processing CreateTablet for tablet cc8311a366664f399d9293ab220c6ca5 (DEFAULT_TABLE table=test-workload [id=5b7ebd0edd6c4642a2828858c9701b86]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251027 03:07:54.023672 29739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc8311a366664f399d9293ab220c6ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:54.024161 29823 tablet_bootstrap.cc:492] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6: Bootstrap starting.
I20251027 03:07:54.024186 29661 tablet_service.cc:1505] Processing CreateTablet for tablet 2a3b7fa0729e4809957a2552309adef9 (DEFAULT_TABLE table=test-workload [id=5b7ebd0edd6c4642a2828858c9701b86]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251027 03:07:54.024262 29661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a3b7fa0729e4809957a2552309adef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:54.024566 29823 tablet_bootstrap.cc:654] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:54.024803 29824 tablet_bootstrap.cc:492] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d: Bootstrap starting.
I20251027 03:07:54.025065 29823 tablet_bootstrap.cc:492] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6: No bootstrap required, opened a new log
I20251027 03:07:54.025104 29823 ts_tablet_manager.cc:1403] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:54.025193 29824 tablet_bootstrap.cc:654] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:54.025241 29823 raft_consensus.cc:359] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f6f2aeeceb4b46a192d86970b401d6" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 46841 } }
I20251027 03:07:54.025281 29823 raft_consensus.cc:385] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:54.025298 29823 raft_consensus.cc:740] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3f6f2aeeceb4b46a192d86970b401d6, State: Initialized, Role: FOLLOWER
I20251027 03:07:54.025346 29823 consensus_queue.cc:260] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f6f2aeeceb4b46a192d86970b401d6" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 46841 } }
I20251027 03:07:54.025378 29823 raft_consensus.cc:399] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:54.025395 29823 raft_consensus.cc:493] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:54.025414 29823 raft_consensus.cc:3060] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:54.025606 29827 tablet_bootstrap.cc:492] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90: Bootstrap starting.
I20251027 03:07:54.025974 29827 tablet_bootstrap.cc:654] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:54.026005 29823 raft_consensus.cc:515] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f6f2aeeceb4b46a192d86970b401d6" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 46841 } }
I20251027 03:07:54.026075 29823 leader_election.cc:304] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3f6f2aeeceb4b46a192d86970b401d6; no voters:
I20251027 03:07:54.026187 29823 leader_election.cc:290] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:54.026224 29828 raft_consensus.cc:2804] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:54.026327 29828 raft_consensus.cc:697] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 1 LEADER]: Becoming Leader. State: Replica: a3f6f2aeeceb4b46a192d86970b401d6, State: Running, Role: LEADER
I20251027 03:07:54.026325 29823 ts_tablet_manager.cc:1434] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:54.026533 29828 consensus_queue.cc:237] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f6f2aeeceb4b46a192d86970b401d6" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 46841 } }
I20251027 03:07:54.026746 29824 tablet_bootstrap.cc:492] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d: No bootstrap required, opened a new log
I20251027 03:07:54.026809 29824 ts_tablet_manager.cc:1403] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:54.026998 29824 raft_consensus.cc:359] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6aacdcded44c93b5af1fd084a75c8d" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35985 } }
I20251027 03:07:54.027093 29824 raft_consensus.cc:385] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:54.027132 29824 raft_consensus.cc:740] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6aacdcded44c93b5af1fd084a75c8d, State: Initialized, Role: FOLLOWER
I20251027 03:07:54.027123 29335 catalog_manager.cc:5649] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3f6f2aeeceb4b46a192d86970b401d6 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "a3f6f2aeeceb4b46a192d86970b401d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3f6f2aeeceb4b46a192d86970b401d6" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 46841 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:54.027233 29824 consensus_queue.cc:260] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6aacdcded44c93b5af1fd084a75c8d" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35985 } }
I20251027 03:07:54.027271 29824 raft_consensus.cc:399] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:54.027289 29824 raft_consensus.cc:493] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:54.027309 29824 raft_consensus.cc:3060] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:54.027333 29827 tablet_bootstrap.cc:492] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90: No bootstrap required, opened a new log
I20251027 03:07:54.027405 29827 ts_tablet_manager.cc:1403] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:54.027976 29824 raft_consensus.cc:515] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6aacdcded44c93b5af1fd084a75c8d" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35985 } }
I20251027 03:07:54.028275 29824 leader_election.cc:304] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c6aacdcded44c93b5af1fd084a75c8d; no voters:
I20251027 03:07:54.028313 29827 raft_consensus.cc:359] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71abed830bde4ebda19c38634a581a90" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33773 } }
I20251027 03:07:54.028366 29827 raft_consensus.cc:385] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:54.028391 29827 raft_consensus.cc:740] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71abed830bde4ebda19c38634a581a90, State: Initialized, Role: FOLLOWER
I20251027 03:07:54.028431 29827 consensus_queue.cc:260] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71abed830bde4ebda19c38634a581a90" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33773 } }
I20251027 03:07:54.028466 29827 raft_consensus.cc:399] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:54.028487 29827 raft_consensus.cc:493] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:54.028512 29827 raft_consensus.cc:3060] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:54.028616 29824 leader_election.cc:290] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:54.028802 29824 ts_tablet_manager.cc:1434] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:54.029173 29827 raft_consensus.cc:515] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71abed830bde4ebda19c38634a581a90" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33773 } }
I20251027 03:07:54.029237 29827 leader_election.cc:304] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71abed830bde4ebda19c38634a581a90; no voters:
I20251027 03:07:54.029349 29827 leader_election.cc:290] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:54.029465 29836 raft_consensus.cc:2804] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:54.029469 29827 ts_tablet_manager.cc:1434] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:54.029635 29836 raft_consensus.cc:697] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 1 LEADER]: Becoming Leader. State: Replica: 71abed830bde4ebda19c38634a581a90, State: Running, Role: LEADER
I20251027 03:07:54.029685 29836 consensus_queue.cc:237] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71abed830bde4ebda19c38634a581a90" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33773 } }
I20251027 03:07:54.029858 29834 raft_consensus.cc:2804] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:54.030159 29335 catalog_manager.cc:5649] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71abed830bde4ebda19c38634a581a90 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "71abed830bde4ebda19c38634a581a90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71abed830bde4ebda19c38634a581a90" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33773 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:54.030524 29834 raft_consensus.cc:697] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 1 LEADER]: Becoming Leader. State: Replica: 7c6aacdcded44c93b5af1fd084a75c8d, State: Running, Role: LEADER
I20251027 03:07:54.030567 29834 consensus_queue.cc:237] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6aacdcded44c93b5af1fd084a75c8d" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35985 } }
I20251027 03:07:54.031167 29334 catalog_manager.cc:5649] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6aacdcded44c93b5af1fd084a75c8d (127.27.151.67). New cstate: current_term: 1 leader_uuid: "7c6aacdcded44c93b5af1fd084a75c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c6aacdcded44c93b5af1fd084a75c8d" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35985 } health_report { overall_health: HEALTHY } } }
I20251027 03:07:54.035949 28253 master.cc:561] Master@127.27.151.126:46131 shutting down...
I20251027 03:07:54.037784 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:54.037839 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:54.037855 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 03699e34e04f4250aa533c9b43777b86: stopping tablet replica
I20251027 03:07:54.050014 28253 master.cc:583] Master@127.27.151.126:46131 shutdown complete.
W20251027 03:07:55.029469 29619 heartbeater.cc:646] Failed to heartbeat to 127.27.151.126:46131 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:55.533252 29846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03699e34e04f4250aa533c9b43777b86)
I20251027 03:07:55.533341 29846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:55.533524 29846 leader_election.cc:290] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131), c5ac390d1adc47d48e4c54812ba085d3 (127.27.151.124:39135)
I20251027 03:07:55.533682 29847 raft_consensus.cc:493] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 03699e34e04f4250aa533c9b43777b86)
I20251027 03:07:55.533763 29477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0815b60b50e44901ad5ec34e006374d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c5ac390d1adc47d48e4c54812ba085d3" is_pre_election: true
I20251027 03:07:55.533771 29847 raft_consensus.cc:515] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:55.533986 29477 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0815b60b50e44901ad5ec34e006374d7 in term 1.
I20251027 03:07:55.534037 29847 leader_election.cc:290] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131), 0815b60b50e44901ad5ec34e006374d7 (127.27.151.125:45191)
I20251027 03:07:55.534196 29387 leader_election.cc:304] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0815b60b50e44901ad5ec34e006374d7, c5ac390d1adc47d48e4c54812ba085d3; no voters:
I20251027 03:07:55.534246 29411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ac390d1adc47d48e4c54812ba085d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0815b60b50e44901ad5ec34e006374d7" is_pre_election: true
I20251027 03:07:55.534283 29846 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 03:07:55.534308 29411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ac390d1adc47d48e4c54812ba085d3 in term 1.
I20251027 03:07:55.534328 29846 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 03699e34e04f4250aa533c9b43777b86)
I20251027 03:07:55.534348 29846 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 1 FOLLOWER]: Advancing to term 2
W20251027 03:07:55.534374 29384 leader_election.cc:336] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:55.534447 29452 leader_election.cc:304] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0815b60b50e44901ad5ec34e006374d7, c5ac390d1adc47d48e4c54812ba085d3; no voters:
I20251027 03:07:55.534533 29847 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 03:07:55.534595 29847 raft_consensus.cc:493] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 03699e34e04f4250aa533c9b43777b86)
I20251027 03:07:55.534623 29847 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 1 FOLLOWER]: Advancing to term 2
W20251027 03:07:55.534639 29450 leader_election.cc:336] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:55.535192 29846 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:55.535295 29847 raft_consensus.cc:515] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:55.535426 29847 leader_election.cc:290] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 2 election: Requested vote from peers 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131), 0815b60b50e44901ad5ec34e006374d7 (127.27.151.125:45191)
I20251027 03:07:55.535488 29477 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0815b60b50e44901ad5ec34e006374d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c5ac390d1adc47d48e4c54812ba085d3"
I20251027 03:07:55.535328 29846 leader_election.cc:290] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [CANDIDATE]: Term 2 election: Requested vote from peers 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131), c5ac390d1adc47d48e4c54812ba085d3 (127.27.151.124:39135)
I20251027 03:07:55.535540 29411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ac390d1adc47d48e4c54812ba085d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0815b60b50e44901ad5ec34e006374d7"
I20251027 03:07:55.535552 29477 raft_consensus.cc:2393] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0815b60b50e44901ad5ec34e006374d7 in current term 2: Already voted for candidate c5ac390d1adc47d48e4c54812ba085d3 in this term.
I20251027 03:07:55.535614 29411 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ac390d1adc47d48e4c54812ba085d3 in current term 2: Already voted for candidate 0815b60b50e44901ad5ec34e006374d7 in this term.
W20251027 03:07:55.535741 29450 leader_election.cc:336] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:55.535786 29450 leader_election.cc:304] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ac390d1adc47d48e4c54812ba085d3; no voters: 03699e34e04f4250aa533c9b43777b86, 0815b60b50e44901ad5ec34e006374d7
I20251027 03:07:55.535851 29847 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251027 03:07:55.535933 29384 leader_election.cc:336] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:55.535961 29384 leader_election.cc:304] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0815b60b50e44901ad5ec34e006374d7; no voters: 03699e34e04f4250aa533c9b43777b86, c5ac390d1adc47d48e4c54812ba085d3
I20251027 03:07:55.536011 29846 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251027 03:07:57.276970 29858 raft_consensus.cc:493] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:57.277078 29858 raft_consensus.cc:515] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:57.277236 29858 leader_election.cc:290] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131), 0815b60b50e44901ad5ec34e006374d7 (127.27.151.125:45191)
I20251027 03:07:57.277467 29411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ac390d1adc47d48e4c54812ba085d3" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0815b60b50e44901ad5ec34e006374d7" is_pre_election: true
I20251027 03:07:57.277563 29411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ac390d1adc47d48e4c54812ba085d3 in term 2.
W20251027 03:07:57.277621 29450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20251027 03:07:57.277717 29452 leader_election.cc:304] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0815b60b50e44901ad5ec34e006374d7, c5ac390d1adc47d48e4c54812ba085d3; no voters:
I20251027 03:07:57.277820 29858 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251027 03:07:57.277866 29858 raft_consensus.cc:493] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:57.277889 29858 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 2 FOLLOWER]: Advancing to term 3
W20251027 03:07:57.278059 29450 leader_election.cc:336] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:57.278527 29858 raft_consensus.cc:515] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:57.278632 29858 leader_election.cc:290] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 3 election: Requested vote from peers 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131), 0815b60b50e44901ad5ec34e006374d7 (127.27.151.125:45191)
I20251027 03:07:57.278776 29411 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ac390d1adc47d48e4c54812ba085d3" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0815b60b50e44901ad5ec34e006374d7"
I20251027 03:07:57.278839 29411 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 2 FOLLOWER]: Advancing to term 3
W20251027 03:07:57.279073 29450 leader_election.cc:336] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111)
I20251027 03:07:57.279475 29411 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ac390d1adc47d48e4c54812ba085d3 in term 3.
I20251027 03:07:57.279613 29452 leader_election.cc:304] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0815b60b50e44901ad5ec34e006374d7, c5ac390d1adc47d48e4c54812ba085d3; no voters: 03699e34e04f4250aa533c9b43777b86
I20251027 03:07:57.279698 29858 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:07:57.279816 29858 raft_consensus.cc:697] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 3 LEADER]: Becoming Leader. State: Replica: c5ac390d1adc47d48e4c54812ba085d3, State: Running, Role: LEADER
I20251027 03:07:57.279873 29858 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } }
I20251027 03:07:57.280128 29861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ac390d1adc47d48e4c54812ba085d3. Latest consensus state: current_term: 3 leader_uuid: "c5ac390d1adc47d48e4c54812ba085d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:57.280225 29861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:57.280361 29863 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:07:57.280573 29863 catalog_manager.cc:679] Loaded metadata for table test-workload [id=5b7ebd0edd6c4642a2828858c9701b86]
I20251027 03:07:57.280684 29863 tablet_loader.cc:96] loaded metadata for tablet 019ac74f34d64573ae4ae976ee2d5bb9 (table test-workload [id=5b7ebd0edd6c4642a2828858c9701b86])
I20251027 03:07:57.280731 29863 tablet_loader.cc:96] loaded metadata for tablet 2a3b7fa0729e4809957a2552309adef9 (table test-workload [id=5b7ebd0edd6c4642a2828858c9701b86])
I20251027 03:07:57.280787 29863 tablet_loader.cc:96] loaded metadata for tablet cc8311a366664f399d9293ab220c6ca5 (table test-workload [id=5b7ebd0edd6c4642a2828858c9701b86])
I20251027 03:07:57.280820 29863 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:07:57.280879 29863 catalog_manager.cc:1269] Loaded cluster ID: 2a11fecd9bb6485eb76a1fe3902a9041
I20251027 03:07:57.280905 29863 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:07:57.281064 29863 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:07:57.281133 29863 catalog_manager.cc:6033] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: Loaded TSK: 0
I20251027 03:07:57.281235 29863 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:07:57.645016 29518 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:07:57.645117 29518 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251027 03:07:57.747424 29450 consensus_peers.cc:597] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 -> Peer 03699e34e04f4250aa533c9b43777b86 (127.27.151.126:46131): Couldn't send request to peer 03699e34e04f4250aa533c9b43777b86. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.126:46131: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:07:57.869905 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:07:57.872531 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:57.872642 28253 tablet_replica.cc:333] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6: stopping tablet replica
I20251027 03:07:57.872679 28253 raft_consensus.cc:2243] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:57.872730 28253 raft_consensus.cc:2272] T 019ac74f34d64573ae4ae976ee2d5bb9 P a3f6f2aeeceb4b46a192d86970b401d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:57.882294 29411 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 3 FOLLOWER]: Refusing update from remote peer c5ac390d1adc47d48e4c54812ba085d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20251027 03:07:57.882508 29861 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20251027 03:07:57.882954 29864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ac390d1adc47d48e4c54812ba085d3. Latest consensus state: current_term: 3 leader_uuid: "c5ac390d1adc47d48e4c54812ba085d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:57.883014 29864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:57.883419 29866 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c5ac390d1adc47d48e4c54812ba085d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:57.883495 29866 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:57.883679 29864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c5ac390d1adc47d48e4c54812ba085d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03699e34e04f4250aa533c9b43777b86" member_type: VOTER last_known_addr { host: "127.27.151.126" port: 46131 } } peers { permanent_uuid: "0815b60b50e44901ad5ec34e006374d7" member_type: VOTER last_known_addr { host: "127.27.151.125" port: 45191 } } peers { permanent_uuid: "c5ac390d1adc47d48e4c54812ba085d3" member_type: VOTER last_known_addr { host: "127.27.151.124" port: 39135 } } }
I20251027 03:07:57.883751 29864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 03:07:57.884038 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:07:57.884846 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:07:57.887264 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:57.887367 28253 tablet_replica.cc:333] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90: stopping tablet replica
I20251027 03:07:57.887413 28253 raft_consensus.cc:2243] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:57.887455 28253 raft_consensus.cc:2272] T 2a3b7fa0729e4809957a2552309adef9 P 71abed830bde4ebda19c38634a581a90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:57.898741 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:07:57.899411 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:07:57.901791 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:07:57.901878 28253 tablet_replica.cc:333] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d: stopping tablet replica
I20251027 03:07:57.901928 28253 raft_consensus.cc:2243] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:07:57.901966 28253 raft_consensus.cc:2272] T cc8311a366664f399d9293ab220c6ca5 P 7c6aacdcded44c93b5af1fd084a75c8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:57.913206 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:07:57.913870 28253 master.cc:561] Master@127.27.151.125:45191 shutting down...
I20251027 03:07:57.915701 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:07:57.915752 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:57.915792 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0815b60b50e44901ad5ec34e006374d7: stopping tablet replica
I20251027 03:07:57.927278 28253 master.cc:583] Master@127.27.151.125:45191 shutdown complete.
I20251027 03:07:57.928220 28253 master.cc:561] Master@127.27.151.124:39135 shutting down...
I20251027 03:07:57.930474 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 3 LEADER]: Raft consensus shutting down.
I20251027 03:07:57.930553 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:07:57.930629 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P c5ac390d1adc47d48e4c54812ba085d3: stopping tablet replica
I20251027 03:07:57.932133 28253 master.cc:583] Master@127.27.151.124:39135 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6355 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20251027 03:07:57.937028 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:34865
I20251027 03:07:57.937145 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:57.938187 29871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:57.938213 29873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:57.938295 29870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:57.938501 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:57.938585 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:57.938616 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:57.938638 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534477938639 us; error 0 us; skew 500 ppm
I20251027 03:07:57.939136 28253 webserver.cc:492] Webserver started at http://127.27.151.126:36327/ using document root <none> and password file <none>
I20251027 03:07:57.939198 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:57.939229 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:57.939260 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:57.939476 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "50407195c13647f499e773e77c1a2b47"
format_stamp: "Formatted at 2025-10-27 03:07:57 on dist-test-slave-4cc8"
I20251027 03:07:57.940325 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:57.940727 29878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:57.940866 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:57.940907 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "50407195c13647f499e773e77c1a2b47"
format_stamp: "Formatted at 2025-10-27 03:07:57 on dist-test-slave-4cc8"
I20251027 03:07:57.940951 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:57.960724 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:57.960887 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:57.963171 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:34865
I20251027 03:07:57.965930 29930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:34865 every 8 connection(s)
I20251027 03:07:57.966081 29931 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:57.966921 29931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47: Bootstrap starting.
I20251027 03:07:57.967162 29931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:57.967551 29931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47: No bootstrap required, opened a new log
I20251027 03:07:57.967657 29931 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50407195c13647f499e773e77c1a2b47" member_type: VOTER }
I20251027 03:07:57.967700 29931 raft_consensus.cc:385] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:57.967711 29931 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50407195c13647f499e773e77c1a2b47, State: Initialized, Role: FOLLOWER
I20251027 03:07:57.967756 29931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50407195c13647f499e773e77c1a2b47" member_type: VOTER }
I20251027 03:07:57.967805 29931 raft_consensus.cc:399] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:07:57.967832 29931 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:07:57.967861 29931 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:57.968319 29931 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50407195c13647f499e773e77c1a2b47" member_type: VOTER }
I20251027 03:07:57.968367 29931 leader_election.cc:304] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50407195c13647f499e773e77c1a2b47; no voters:
I20251027 03:07:57.968454 29931 leader_election.cc:290] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:07:57.968498 29934 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:57.968580 29931 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:07:57.968621 29934 raft_consensus.cc:697] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 1 LEADER]: Becoming Leader. State: Replica: 50407195c13647f499e773e77c1a2b47, State: Running, Role: LEADER
I20251027 03:07:57.968693 29934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50407195c13647f499e773e77c1a2b47" member_type: VOTER }
I20251027 03:07:57.968856 29934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50407195c13647f499e773e77c1a2b47. Latest consensus state: current_term: 1 leader_uuid: "50407195c13647f499e773e77c1a2b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50407195c13647f499e773e77c1a2b47" member_type: VOTER } }
I20251027 03:07:57.968902 29934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:57.969018 29936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50407195c13647f499e773e77c1a2b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50407195c13647f499e773e77c1a2b47" member_type: VOTER } }
I20251027 03:07:57.969074 29936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [sys.catalog]: This master's current role is: LEADER
I20251027 03:07:57.969293 29939 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:07:57.969409 29939 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:07:57.969764 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:07:57.969870 29939 catalog_manager.cc:1357] Generated new cluster ID: 06e771d789334c9a9a4dad10543a7371
I20251027 03:07:57.969904 29939 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:07:57.974221 29939 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:07:57.974572 29939 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:07:57.979709 29939 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47: Generated new TSK 0
I20251027 03:07:57.979768 29939 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:07:57.985272 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:57.986333 29955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:57.986413 29953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:57.986408 29952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:57.986486 28253 server_base.cc:1047] running on GCE node
I20251027 03:07:57.986622 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:57.986645 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:57.986662 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534477986662 us; error 0 us; skew 500 ppm
I20251027 03:07:57.987169 28253 webserver.cc:492] Webserver started at http://127.27.151.65:33281/ using document root <none> and password file <none>
I20251027 03:07:57.987239 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:57.987283 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:57.987340 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:57.987591 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "f73d8bcae95d40c681ef625d0a12f570"
format_stamp: "Formatted at 2025-10-27 03:07:57 on dist-test-slave-4cc8"
I20251027 03:07:57.988426 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:57.988849 29960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:57.988981 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:57.989018 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "f73d8bcae95d40c681ef625d0a12f570"
format_stamp: "Formatted at 2025-10-27 03:07:57 on dist-test-slave-4cc8"
I20251027 03:07:57.989074 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:57.996459 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:57.996630 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:57.996762 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:57.996919 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:57.996945 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:57.996975 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:57.997001 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:57.999486 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:41283
I20251027 03:07:57.999533 30023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:41283 every 8 connection(s)
I20251027 03:07:57.999810 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:58.000805 30029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:58.000850 30028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:58.001044 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:58.001330 30031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:58.001492 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:58.001524 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:58.001541 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534478001541 us; error 0 us; skew 500 ppm
I20251027 03:07:58.002089 28253 webserver.cc:492] Webserver started at http://127.27.151.66:33663/ using document root <none> and password file <none>
I20251027 03:07:58.002166 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:58.002213 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:58.002259 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:58.002530 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "98077bc27d8d48dab195ae5172123605"
format_stamp: "Formatted at 2025-10-27 03:07:57 on dist-test-slave-4cc8"
I20251027 03:07:58.003340 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:58.003728 30036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.003858 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.003904 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "98077bc27d8d48dab195ae5172123605"
format_stamp: "Formatted at 2025-10-27 03:07:57 on dist-test-slave-4cc8"
I20251027 03:07:58.003939 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:58.005971 30024 heartbeater.cc:344] Connected to a master server at 127.27.151.126:34865
I20251027 03:07:58.006031 30024 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:58.006140 30024 heartbeater.cc:507] Master 127.27.151.126:34865 requested a full tablet report, sending...
I20251027 03:07:58.006368 29895 ts_manager.cc:194] Registered new tserver with Master: f73d8bcae95d40c681ef625d0a12f570 (127.27.151.65:41283)
I20251027 03:07:58.007121 29895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34864
I20251027 03:07:58.032287 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:58.032481 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:58.032598 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:58.032781 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:58.032809 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.032842 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:58.032877 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.035517 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:38113
I20251027 03:07:58.035580 30099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:38113 every 8 connection(s)
I20251027 03:07:58.035830 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:07:58.036983 30104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:07:58.036991 30103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:58.037189 28253 server_base.cc:1047] running on GCE node
W20251027 03:07:58.037288 30106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:07:58.037436 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:07:58.037465 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:07:58.037482 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534478037482 us; error 0 us; skew 500 ppm
I20251027 03:07:58.038061 28253 webserver.cc:492] Webserver started at http://127.27.151.67:43503/ using document root <none> and password file <none>
I20251027 03:07:58.038136 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:07:58.038178 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:07:58.038223 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:07:58.038482 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "5ec38df29de54cc1b2a772879dc24827"
format_stamp: "Formatted at 2025-10-27 03:07:58 on dist-test-slave-4cc8"
I20251027 03:07:58.039089 30100 heartbeater.cc:344] Connected to a master server at 127.27.151.126:34865
I20251027 03:07:58.039155 30100 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:58.039268 30100 heartbeater.cc:507] Master 127.27.151.126:34865 requested a full tablet report, sending...
I20251027 03:07:58.039498 29895 ts_manager.cc:194] Registered new tserver with Master: 98077bc27d8d48dab195ae5172123605 (127.27.151.66:38113)
I20251027 03:07:58.039603 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:07:58.039865 29895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34866
I20251027 03:07:58.040079 30111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.040234 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.040277 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "5ec38df29de54cc1b2a772879dc24827"
format_stamp: "Formatted at 2025-10-27 03:07:58 on dist-test-slave-4cc8"
I20251027 03:07:58.040324 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:07:58.047170 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:07:58.047361 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:07:58.047497 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:07:58.047653 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:07:58.047679 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.047708 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:07:58.047734 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:58.050727 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:38203
I20251027 03:07:58.050792 30174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:38203 every 8 connection(s)
I20251027 03:07:58.053208 30175 heartbeater.cc:344] Connected to a master server at 127.27.151.126:34865
I20251027 03:07:58.053253 30175 heartbeater.cc:461] Registering TS with master...
I20251027 03:07:58.053333 30175 heartbeater.cc:507] Master 127.27.151.126:34865 requested a full tablet report, sending...
I20251027 03:07:58.053486 29895 ts_manager.cc:194] Registered new tserver with Master: 5ec38df29de54cc1b2a772879dc24827 (127.27.151.67:38203)
I20251027 03:07:58.053771 29895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34880
I20251027 03:07:58.054168 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003265022s
I20251027 03:07:59.007601 30024 heartbeater.cc:499] Master 127.27.151.126:34865 was elected leader, sending a full tablet report...
I20251027 03:07:59.040570 30100 heartbeater.cc:499] Master 127.27.151.126:34865 was elected leader, sending a full tablet report...
I20251027 03:07:59.054193 30175 heartbeater.cc:499] Master 127.27.151.126:34865 was elected leader, sending a full tablet report...
I20251027 03:07:59.078769 28253 test_util.cc:276] Using random seed: 887264623
I20251027 03:07:59.083729 29895 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34886:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:07:59.083864 29895 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:07:59.088819 30139 tablet_service.cc:1505] Processing CreateTablet for tablet 7e06ed0bdead4cd79b5e8c4b03de4383 (DEFAULT_TABLE table=test-workload [id=343623eeac934b039a64a4e2da7fc531]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251027 03:07:59.088944 30139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e06ed0bdead4cd79b5e8c4b03de4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:59.089476 30064 tablet_service.cc:1505] Processing CreateTablet for tablet 7e06ed0bdead4cd79b5e8c4b03de4383 (DEFAULT_TABLE table=test-workload [id=343623eeac934b039a64a4e2da7fc531]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251027 03:07:59.089494 30063 tablet_service.cc:1505] Processing CreateTablet for tablet c445f1a08540406587ac3987c435447c (DEFAULT_TABLE table=test-workload [id=343623eeac934b039a64a4e2da7fc531]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251027 03:07:59.089581 30064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e06ed0bdead4cd79b5e8c4b03de4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:59.089689 30063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c445f1a08540406587ac3987c435447c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:59.090320 30138 tablet_service.cc:1505] Processing CreateTablet for tablet c445f1a08540406587ac3987c435447c (DEFAULT_TABLE table=test-workload [id=343623eeac934b039a64a4e2da7fc531]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251027 03:07:59.090394 30138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c445f1a08540406587ac3987c435447c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:59.090783 30196 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: Bootstrap starting.
I20251027 03:07:59.090885 29988 tablet_service.cc:1505] Processing CreateTablet for tablet 7e06ed0bdead4cd79b5e8c4b03de4383 (DEFAULT_TABLE table=test-workload [id=343623eeac934b039a64a4e2da7fc531]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251027 03:07:59.090966 29988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e06ed0bdead4cd79b5e8c4b03de4383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:59.091202 30196 tablet_bootstrap.cc:654] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:59.091758 30196 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: No bootstrap required, opened a new log
I20251027 03:07:59.091801 30196 ts_tablet_manager.cc:1403] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:59.091930 30196 raft_consensus.cc:359] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.091986 30196 raft_consensus.cc:385] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:59.092001 30196 raft_consensus.cc:740] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98077bc27d8d48dab195ae5172123605, State: Initialized, Role: FOLLOWER
I20251027 03:07:59.092039 30196 consensus_queue.cc:260] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.092108 29987 tablet_service.cc:1505] Processing CreateTablet for tablet c445f1a08540406587ac3987c435447c (DEFAULT_TABLE table=test-workload [id=343623eeac934b039a64a4e2da7fc531]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251027 03:07:59.092216 30198 tablet_bootstrap.cc:492] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: Bootstrap starting.
I20251027 03:07:59.092243 30196 ts_tablet_manager.cc:1434] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:07:59.092221 29987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c445f1a08540406587ac3987c435447c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:07:59.092432 30196 tablet_bootstrap.cc:492] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605: Bootstrap starting.
I20251027 03:07:59.092911 30198 tablet_bootstrap.cc:654] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:59.092911 30196 tablet_bootstrap.cc:654] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:59.093480 30198 tablet_bootstrap.cc:492] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: No bootstrap required, opened a new log
I20251027 03:07:59.093521 30198 ts_tablet_manager.cc:1403] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:59.092361 30195 tablet_bootstrap.cc:492] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827: Bootstrap starting.
I20251027 03:07:59.093743 30198 raft_consensus.cc:359] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.093799 30198 raft_consensus.cc:385] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:59.093853 30198 raft_consensus.cc:740] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73d8bcae95d40c681ef625d0a12f570, State: Initialized, Role: FOLLOWER
I20251027 03:07:59.093919 30198 consensus_queue.cc:260] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.093967 30195 tablet_bootstrap.cc:654] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:59.094133 30198 ts_tablet_manager.cc:1434] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:59.094255 30198 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: Bootstrap starting.
I20251027 03:07:59.094503 30195 tablet_bootstrap.cc:492] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827: No bootstrap required, opened a new log
I20251027 03:07:59.094558 30195 ts_tablet_manager.cc:1403] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:59.094689 30198 tablet_bootstrap.cc:654] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:59.094704 30195 raft_consensus.cc:359] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.094790 30195 raft_consensus.cc:385] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:59.094821 30195 raft_consensus.cc:740] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec38df29de54cc1b2a772879dc24827, State: Initialized, Role: FOLLOWER
I20251027 03:07:59.094870 30195 consensus_queue.cc:260] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.095026 30195 ts_tablet_manager.cc:1434] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:07:59.095113 30195 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827: Bootstrap starting.
I20251027 03:07:59.095273 30196 tablet_bootstrap.cc:492] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605: No bootstrap required, opened a new log
I20251027 03:07:59.095327 30196 ts_tablet_manager.cc:1403] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:07:59.095494 30195 tablet_bootstrap.cc:654] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827: Neither blocks nor log segments found. Creating new log.
I20251027 03:07:59.095494 30196 raft_consensus.cc:359] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.095577 30196 raft_consensus.cc:385] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:59.095647 30196 raft_consensus.cc:740] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98077bc27d8d48dab195ae5172123605, State: Initialized, Role: FOLLOWER
I20251027 03:07:59.095711 30196 consensus_queue.cc:260] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.095853 30196 ts_tablet_manager.cc:1434] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:59.096113 30195 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827: No bootstrap required, opened a new log
I20251027 03:07:59.096210 30195 ts_tablet_manager.cc:1403] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:07:59.096231 30198 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: No bootstrap required, opened a new log
I20251027 03:07:59.096273 30198 ts_tablet_manager.cc:1403] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:07:59.096387 30195 raft_consensus.cc:359] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.096426 30195 raft_consensus.cc:385] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:59.096482 30195 raft_consensus.cc:740] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec38df29de54cc1b2a772879dc24827, State: Initialized, Role: FOLLOWER
I20251027 03:07:59.096432 30198 raft_consensus.cc:359] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.096519 30198 raft_consensus.cc:385] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:07:59.096544 30198 raft_consensus.cc:740] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73d8bcae95d40c681ef625d0a12f570, State: Initialized, Role: FOLLOWER
I20251027 03:07:59.096530 30195 consensus_queue.cc:260] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.096588 30198 consensus_queue.cc:260] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.096613 30195 ts_tablet_manager.cc:1434] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:59.096657 30198 ts_tablet_manager.cc:1434] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:07:59.211216 30202 raft_consensus.cc:493] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:59.211309 30202 raft_consensus.cc:515] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.211464 30202 leader_election.cc:290] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98077bc27d8d48dab195ae5172123605 (127.27.151.66:38113), 5ec38df29de54cc1b2a772879dc24827 (127.27.151.67:38203)
I20251027 03:07:59.214030 30149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec38df29de54cc1b2a772879dc24827" is_pre_election: true
I20251027 03:07:59.214107 30149 raft_consensus.cc:2468] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 0.
I20251027 03:07:59.214047 30074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98077bc27d8d48dab195ae5172123605" is_pre_election: true
I20251027 03:07:59.214146 30074 raft_consensus.cc:2468] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 0.
I20251027 03:07:59.214250 29964 leader_election.cc:304] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec38df29de54cc1b2a772879dc24827, f73d8bcae95d40c681ef625d0a12f570; no voters:
I20251027 03:07:59.214334 30202 raft_consensus.cc:2804] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:59.214370 30202 raft_consensus.cc:493] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:59.214393 30202 raft_consensus.cc:3060] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:59.214926 30202 raft_consensus.cc:515] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.215039 30202 leader_election.cc:290] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 election: Requested vote from peers 98077bc27d8d48dab195ae5172123605 (127.27.151.66:38113), 5ec38df29de54cc1b2a772879dc24827 (127.27.151.67:38203)
I20251027 03:07:59.215167 30074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98077bc27d8d48dab195ae5172123605"
I20251027 03:07:59.215198 30149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec38df29de54cc1b2a772879dc24827"
I20251027 03:07:59.215243 30074 raft_consensus.cc:3060] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:59.215258 30149 raft_consensus.cc:3060] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:59.215904 30074 raft_consensus.cc:2468] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 1.
I20251027 03:07:59.215904 30149 raft_consensus.cc:2468] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 1.
I20251027 03:07:59.216080 29964 leader_election.cc:304] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec38df29de54cc1b2a772879dc24827, f73d8bcae95d40c681ef625d0a12f570; no voters:
I20251027 03:07:59.216176 30202 raft_consensus.cc:2804] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:59.216305 30202 raft_consensus.cc:697] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Becoming Leader. State: Replica: f73d8bcae95d40c681ef625d0a12f570, State: Running, Role: LEADER
I20251027 03:07:59.216357 30202 consensus_queue.cc:237] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.216826 29894 catalog_manager.cc:5649] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 reported cstate change: term changed from 0 to 1, leader changed from <none> to f73d8bcae95d40c681ef625d0a12f570 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "f73d8bcae95d40c681ef625d0a12f570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:59.396921 30202 raft_consensus.cc:493] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:07:59.397017 30202 raft_consensus.cc:515] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.397161 30202 leader_election.cc:290] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98077bc27d8d48dab195ae5172123605 (127.27.151.66:38113), 5ec38df29de54cc1b2a772879dc24827 (127.27.151.67:38203)
I20251027 03:07:59.397356 30149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c445f1a08540406587ac3987c435447c" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec38df29de54cc1b2a772879dc24827" is_pre_election: true
I20251027 03:07:59.397426 30149 raft_consensus.cc:2468] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 0.
I20251027 03:07:59.397392 30074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c445f1a08540406587ac3987c435447c" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98077bc27d8d48dab195ae5172123605" is_pre_election: true
I20251027 03:07:59.397495 30074 raft_consensus.cc:2468] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 0.
I20251027 03:07:59.397558 29964 leader_election.cc:304] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec38df29de54cc1b2a772879dc24827, f73d8bcae95d40c681ef625d0a12f570; no voters:
I20251027 03:07:59.397675 30202 raft_consensus.cc:2804] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:07:59.397711 30202 raft_consensus.cc:493] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:07:59.397723 30202 raft_consensus.cc:3060] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:59.398339 30202 raft_consensus.cc:515] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.398444 30202 leader_election.cc:290] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 election: Requested vote from peers 98077bc27d8d48dab195ae5172123605 (127.27.151.66:38113), 5ec38df29de54cc1b2a772879dc24827 (127.27.151.67:38203)
I20251027 03:07:59.398599 30149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c445f1a08540406587ac3987c435447c" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec38df29de54cc1b2a772879dc24827"
I20251027 03:07:59.398608 30074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c445f1a08540406587ac3987c435447c" candidate_uuid: "f73d8bcae95d40c681ef625d0a12f570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98077bc27d8d48dab195ae5172123605"
I20251027 03:07:59.398658 30149 raft_consensus.cc:3060] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:59.398669 30074 raft_consensus.cc:3060] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:07:59.399183 30074 raft_consensus.cc:2468] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 1.
I20251027 03:07:59.399188 30149 raft_consensus.cc:2468] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f73d8bcae95d40c681ef625d0a12f570 in term 1.
I20251027 03:07:59.399339 29964 leader_election.cc:304] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec38df29de54cc1b2a772879dc24827, f73d8bcae95d40c681ef625d0a12f570; no voters:
I20251027 03:07:59.399434 30202 raft_consensus.cc:2804] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:07:59.399497 30202 raft_consensus.cc:697] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Becoming Leader. State: Replica: f73d8bcae95d40c681ef625d0a12f570, State: Running, Role: LEADER
I20251027 03:07:59.399549 30202 consensus_queue.cc:237] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:07:59.400068 29894 catalog_manager.cc:5649] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 reported cstate change: term changed from 0 to 1, leader changed from <none> to f73d8bcae95d40c681ef625d0a12f570 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "f73d8bcae95d40c681ef625d0a12f570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20251027 03:07:59.678804 30202 consensus_queue.cc:1048] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:59.680399 30202 consensus_queue.cc:1048] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:59.805387 30202 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:59.806779 30220 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:07:59.969834 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:07:59.972788 29998 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383"
dest_uuid: "f73d8bcae95d40c681ef625d0a12f570"
mode: GRACEFUL
new_leader_uuid: "98077bc27d8d48dab195ae5172123605"
from {username='slave'} at 127.0.0.1:45106
I20251027 03:07:59.972874 29998 raft_consensus.cc:606] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Received request to transfer leadership to TS 98077bc27d8d48dab195ae5172123605
I20251027 03:07:59.973021 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:07:59.973065 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:00.181763 30220 raft_consensus.cc:993] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: : Instructing follower 98077bc27d8d48dab195ae5172123605 to start an election
I20251027 03:08:00.181857 30220 raft_consensus.cc:1081] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Signalling peer 98077bc27d8d48dab195ae5172123605 to start an election
I20251027 03:08:00.182037 30074 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383"
dest_uuid: "98077bc27d8d48dab195ae5172123605"
from {username='slave'} at 127.0.0.1:36422
I20251027 03:08:00.182106 30074 raft_consensus.cc:493] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:08:00.182126 30074 raft_consensus.cc:3060] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:08:00.182678 30074 raft_consensus.cc:515] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:08:00.182818 30074 leader_election.cc:290] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [CANDIDATE]: Term 2 election: Requested vote from peers f73d8bcae95d40c681ef625d0a12f570 (127.27.151.65:41283), 5ec38df29de54cc1b2a772879dc24827 (127.27.151.67:38203)
I20251027 03:08:00.185325 29998 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383" candidate_uuid: "98077bc27d8d48dab195ae5172123605" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f73d8bcae95d40c681ef625d0a12f570"
I20251027 03:08:00.185339 30149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e06ed0bdead4cd79b5e8c4b03de4383" candidate_uuid: "98077bc27d8d48dab195ae5172123605" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ec38df29de54cc1b2a772879dc24827"
I20251027 03:08:00.185398 29998 raft_consensus.cc:3055] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:08:00.185501 30149 raft_consensus.cc:3060] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:08:00.185501 29998 raft_consensus.cc:740] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f73d8bcae95d40c681ef625d0a12f570, State: Running, Role: LEADER
I20251027 03:08:00.185592 29998 consensus_queue.cc:260] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:08:00.185676 29998 raft_consensus.cc:3060] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:08:00.186125 30149 raft_consensus.cc:2468] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98077bc27d8d48dab195ae5172123605 in term 2.
I20251027 03:08:00.186252 29998 raft_consensus.cc:2468] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98077bc27d8d48dab195ae5172123605 in term 2.
I20251027 03:08:00.186326 30040 leader_election.cc:304] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec38df29de54cc1b2a772879dc24827, 98077bc27d8d48dab195ae5172123605; no voters:
I20251027 03:08:00.186439 30212 raft_consensus.cc:2804] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:08:00.186569 30212 raft_consensus.cc:697] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 2 LEADER]: Becoming Leader. State: Replica: 98077bc27d8d48dab195ae5172123605, State: Running, Role: LEADER
I20251027 03:08:00.186625 30212 consensus_queue.cc:237] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } }
I20251027 03:08:00.187057 29894 catalog_manager.cc:5649] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 reported cstate change: term changed from 1 to 2, leader changed from f73d8bcae95d40c681ef625d0a12f570 (127.27.151.65) to 98077bc27d8d48dab195ae5172123605 (127.27.151.66). New cstate: current_term: 2 leader_uuid: "98077bc27d8d48dab195ae5172123605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } health_report { overall_health: UNKNOWN } } }
I20251027 03:08:00.688906 30149 raft_consensus.cc:1275] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 2 FOLLOWER]: Refusing update from remote peer 98077bc27d8d48dab195ae5172123605: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:08:00.689148 30232 consensus_queue.cc:1048] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:08:00.690392 29998 raft_consensus.cc:1275] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 2 FOLLOWER]: Refusing update from remote peer 98077bc27d8d48dab195ae5172123605: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:08:00.690624 30238 consensus_queue.cc:1048] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:08:01.973218 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:01.973367 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:01.973409 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:03.973548 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:03.973659 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:03.973678 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:05.973809 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:05.973917 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:05.973937 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:07.974082 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:07.974191 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:07.974213 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:09.974355 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:09.974465 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:09.974483 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:10.486362 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:10.487818 30245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:10.487891 30244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:10.487941 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:10.487954 30247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:10.488256 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:10.488289 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:10.488313 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534490488313 us; error 0 us; skew 500 ppm
I20251027 03:08:10.488970 28253 webserver.cc:492] Webserver started at http://127.27.151.68:34271/ using document root <none> and password file <none>
I20251027 03:08:10.489055 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:10.489104 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:10.489152 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:10.489409 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "5dbfe44566a04947be967d1042a4386a"
format_stamp: "Formatted at 2025-10-27 03:08:10 on dist-test-slave-4cc8"
I20251027 03:08:10.490279 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:08:10.490658 30252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:10.490801 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:10.490851 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "5dbfe44566a04947be967d1042a4386a"
format_stamp: "Formatted at 2025-10-27 03:08:10 on dist-test-slave-4cc8"
I20251027 03:08:10.490893 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:10.497891 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:10.498064 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:10.498214 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:10.498371 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:10.498400 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:10.498418 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:10.498446 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:10.501555 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:46037
I20251027 03:08:10.501605 30315 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:46037 every 8 connection(s)
I20251027 03:08:10.504068 30316 heartbeater.cc:344] Connected to a master server at 127.27.151.126:34865
I20251027 03:08:10.504118 30316 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:10.504220 30316 heartbeater.cc:507] Master 127.27.151.126:34865 requested a full tablet report, sending...
I20251027 03:08:10.504411 29895 ts_manager.cc:194] Registered new tserver with Master: 5dbfe44566a04947be967d1042a4386a (127.27.151.68:46037)
I20251027 03:08:10.504722 29895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43084
I20251027 03:08:10.975252 29998 consensus_queue.cc:237] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } }
I20251027 03:08:10.975844 30074 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:08:10.975971 30149 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:08:10.976073 30325 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:08:10.976229 30324 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:08:10.976907 30324 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5dbfe44566a04947be967d1042a4386a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } } }
I20251027 03:08:10.977119 30074 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5dbfe44566a04947be967d1042a4386a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } } }
I20251027 03:08:10.978116 29895 catalog_manager.cc:5649] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 reported cstate change: config changed from index -1 to 2, NON_VOTER 5dbfe44566a04947be967d1042a4386a (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "f73d8bcae95d40c681ef625d0a12f570" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } } }
I20251027 03:08:10.978600 30149 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5dbfe44566a04947be967d1042a4386a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } } }
W20251027 03:08:10.979254 29964 consensus_peers.cc:597] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 -> Peer 5dbfe44566a04947be967d1042a4386a (127.27.151.68:46037): Couldn't send request to peer 5dbfe44566a04947be967d1042a4386a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c445f1a08540406587ac3987c435447c. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:08:11.385020 30335 ts_tablet_manager.cc:933] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: Initiating tablet copy from peer f73d8bcae95d40c681ef625d0a12f570 (127.27.151.65:41283)
I20251027 03:08:11.385216 30335 tablet_copy_client.cc:323] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:41283
I20251027 03:08:11.387598 30008 tablet_copy_service.cc:140] P f73d8bcae95d40c681ef625d0a12f570: Received BeginTabletCopySession request for tablet c445f1a08540406587ac3987c435447c from peer 5dbfe44566a04947be967d1042a4386a ({username='slave'} at 127.0.0.1:40458)
I20251027 03:08:11.387676 30008 tablet_copy_service.cc:161] P f73d8bcae95d40c681ef625d0a12f570: Beginning new tablet copy session on tablet c445f1a08540406587ac3987c435447c from peer 5dbfe44566a04947be967d1042a4386a at {username='slave'} at 127.0.0.1:40458: session id = 5dbfe44566a04947be967d1042a4386a-c445f1a08540406587ac3987c435447c
I20251027 03:08:11.388448 30008 tablet_copy_source_session.cc:215] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:08:11.388938 30335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c445f1a08540406587ac3987c435447c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:11.390380 30335 tablet_copy_client.cc:806] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: tablet copy: Starting download of 0 data blocks...
I20251027 03:08:11.390523 30335 tablet_copy_client.cc:670] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: tablet copy: Starting download of 1 WAL segments...
I20251027 03:08:11.391136 30335 tablet_copy_client.cc:538] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:08:11.391973 30335 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: Bootstrap starting.
I20251027 03:08:11.393741 30335 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:08:11.393963 30335 tablet_bootstrap.cc:492] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: Bootstrap complete.
I20251027 03:08:11.394061 30335 ts_tablet_manager.cc:1403] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20251027 03:08:11.394234 30335 raft_consensus.cc:359] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } }
I20251027 03:08:11.394320 30335 raft_consensus.cc:740] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5dbfe44566a04947be967d1042a4386a, State: Initialized, Role: LEARNER
I20251027 03:08:11.394362 30335 consensus_queue.cc:260] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: true } }
I20251027 03:08:11.394526 30335 ts_tablet_manager.cc:1434] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:08:11.394582 30316 heartbeater.cc:499] Master 127.27.151.126:34865 was elected leader, sending a full tablet report...
I20251027 03:08:11.394718 30008 tablet_copy_service.cc:342] P f73d8bcae95d40c681ef625d0a12f570: Request end of tablet copy session 5dbfe44566a04947be967d1042a4386a-c445f1a08540406587ac3987c435447c received from {username='slave'} at 127.0.0.1:40458
I20251027 03:08:11.394780 30008 tablet_copy_service.cc:434] P f73d8bcae95d40c681ef625d0a12f570: ending tablet copy session 5dbfe44566a04947be967d1042a4386a-c445f1a08540406587ac3987c435447c on tablet c445f1a08540406587ac3987c435447c with peer 5dbfe44566a04947be967d1042a4386a
I20251027 03:08:11.857065 30289 raft_consensus.cc:1217] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251027 03:08:11.974611 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:11.974736 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:11.974774 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:12.260975 30340 raft_consensus.cc:1064] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: attempting to promote NON_VOTER 5dbfe44566a04947be967d1042a4386a to VOTER
I20251027 03:08:12.261257 30340 consensus_queue.cc:237] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } }
I20251027 03:08:12.261719 30289 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 LEARNER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:08:12.261739 30149 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:08:12.261922 30343 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:08:12.262045 30220 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:08:12.262079 30074 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:08:12.262238 30220 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:08:12.262945 30341 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5dbfe44566a04947be967d1042a4386a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.263103 30149 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5dbfe44566a04947be967d1042a4386a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.263957 30074 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5dbfe44566a04947be967d1042a4386a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.264071 29894 catalog_manager.cc:5649] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 reported cstate change: config changed from index 2 to 3, 5dbfe44566a04947be967d1042a4386a (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f73d8bcae95d40c681ef625d0a12f570" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:08:12.264830 30289 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5dbfe44566a04947be967d1042a4386a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "98077bc27d8d48dab195ae5172123605" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38113 } attrs { replace: true } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.266403 29998 consensus_queue.cc:237] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } }
I20251027 03:08:12.266711 30289 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 FOLLOWER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:08:12.266712 30149 raft_consensus.cc:1275] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Refusing update from remote peer f73d8bcae95d40c681ef625d0a12f570: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:08:12.266891 30343 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:08:12.266948 30220 consensus_queue.cc:1048] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:08:12.267453 30340 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98077bc27d8d48dab195ae5172123605 (127.27.151.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.267575 30149 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98077bc27d8d48dab195ae5172123605 (127.27.151.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.268244 29881 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c445f1a08540406587ac3987c435447c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:08:12.268421 29892 catalog_manager.cc:5649] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 reported cstate change: config changed from index 3 to 4, VOTER 98077bc27d8d48dab195ae5172123605 (127.27.151.66) evicted. New cstate: current_term: 1 leader_uuid: "f73d8bcae95d40c681ef625d0a12f570" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:08:12.268456 30289 raft_consensus.cc:2955] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 98077bc27d8d48dab195ae5172123605 (127.27.151.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f73d8bcae95d40c681ef625d0a12f570" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41283 } } peers { permanent_uuid: "5ec38df29de54cc1b2a772879dc24827" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 38203 } } peers { permanent_uuid: "5dbfe44566a04947be967d1042a4386a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 46037 } attrs { promote: false } } }
I20251027 03:08:12.269136 30064 tablet_service.cc:1552] Processing DeleteTablet for tablet c445f1a08540406587ac3987c435447c with delete_type TABLET_DATA_TOMBSTONED (TS 98077bc27d8d48dab195ae5172123605 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36410
I20251027 03:08:12.269347 30352 tablet_replica.cc:333] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: stopping tablet replica
I20251027 03:08:12.269407 30352 raft_consensus.cc:2243] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:08:12.269492 30352 raft_consensus.cc:2272] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:12.269760 30352 ts_tablet_manager.cc:1916] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:08:12.271108 30352 ts_tablet_manager.cc:1929] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251027 03:08:12.271160 30352 log.cc:1199] T c445f1a08540406587ac3987c435447c P 98077bc27d8d48dab195ae5172123605: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/c445f1a08540406587ac3987c435447c
I20251027 03:08:12.271445 29881 catalog_manager.cc:4985] TS 98077bc27d8d48dab195ae5172123605 (127.27.151.66:38113): tablet c445f1a08540406587ac3987c435447c (table test-workload [id=343623eeac934b039a64a4e2da7fc531]) successfully deleted
I20251027 03:08:13.974939 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:13.975057 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:13.975076 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:15.975299 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:15.975431 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:15.975459 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:17.975600 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:17.975734 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:17.975755 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:19.975910 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:19.976034 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:19.976053 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:21.976274 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:21.976423 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:21.976457 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:23.976683 29949 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:23.976832 29949 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:23.976857 29949 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:24.577122 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:08:24.579909 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:24.580049 28253 tablet_replica.cc:333] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570: stopping tablet replica
I20251027 03:08:24.580117 28253 raft_consensus.cc:2243] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:08:24.580225 28253 raft_consensus.cc:2272] T c445f1a08540406587ac3987c435447c P f73d8bcae95d40c681ef625d0a12f570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.580397 28253 tablet_replica.cc:333] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570: stopping tablet replica
I20251027 03:08:24.580433 28253 raft_consensus.cc:2243] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:08:24.580474 28253 raft_consensus.cc:2272] T 7e06ed0bdead4cd79b5e8c4b03de4383 P f73d8bcae95d40c681ef625d0a12f570 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.592216 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:08:24.593005 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:08:24.595619 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:24.595718 28253 tablet_replica.cc:333] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605: stopping tablet replica
I20251027 03:08:24.595770 28253 raft_consensus.cc:2243] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:08:24.595820 28253 raft_consensus.cc:2272] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 98077bc27d8d48dab195ae5172123605 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.607678 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:08:24.608464 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:08:24.611721 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:24.611816 28253 tablet_replica.cc:333] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827: stopping tablet replica
I20251027 03:08:24.611871 28253 raft_consensus.cc:2243] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:08:24.611925 28253 raft_consensus.cc:2272] T c445f1a08540406587ac3987c435447c P 5ec38df29de54cc1b2a772879dc24827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.612064 28253 tablet_replica.cc:333] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827: stopping tablet replica
I20251027 03:08:24.612092 28253 raft_consensus.cc:2243] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:08:24.612110 28253 raft_consensus.cc:2272] T 7e06ed0bdead4cd79b5e8c4b03de4383 P 5ec38df29de54cc1b2a772879dc24827 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.623512 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:08:24.624305 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:08:24.626956 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:24.627046 28253 tablet_replica.cc:333] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a: stopping tablet replica
I20251027 03:08:24.627094 28253 raft_consensus.cc:2243] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:08:24.627142 28253 raft_consensus.cc:2272] T c445f1a08540406587ac3987c435447c P 5dbfe44566a04947be967d1042a4386a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.638252 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:08:24.638932 28253 master.cc:561] Master@127.27.151.126:34865 shutting down...
I20251027 03:08:24.641178 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:08:24.641235 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:24.641265 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 50407195c13647f499e773e77c1a2b47: stopping tablet replica
I20251027 03:08:24.653259 28253 master.cc:583] Master@127.27.151.126:34865 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26720 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20251027 03:08:24.657842 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:37815
I20251027 03:08:24.657986 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:08:24.659117 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:24.659183 30357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:24.659238 30360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:24.659313 30358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:24.659444 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:24.659478 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:24.659490 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534504659491 us; error 0 us; skew 500 ppm
I20251027 03:08:24.659991 28253 webserver.cc:492] Webserver started at http://127.27.151.126:45109/ using document root <none> and password file <none>
I20251027 03:08:24.660059 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:24.660100 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:24.660219 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:24.660468 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "642ffdcbbac841b591e8b319f0b5dec5"
format_stamp: "Formatted at 2025-10-27 03:08:24 on dist-test-slave-4cc8"
I20251027 03:08:24.661268 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:08:24.661652 30365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:24.661777 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:24.661820 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "642ffdcbbac841b591e8b319f0b5dec5"
format_stamp: "Formatted at 2025-10-27 03:08:24 on dist-test-slave-4cc8"
I20251027 03:08:24.661859 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:24.666569 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:24.666708 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:24.669034 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:37815
I20251027 03:08:24.671157 30417 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:37815 every 8 connection(s)
I20251027 03:08:24.671260 30418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:24.672168 30418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5: Bootstrap starting.
I20251027 03:08:24.672417 30418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5: Neither blocks nor log segments found. Creating new log.
I20251027 03:08:24.672866 30418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5: No bootstrap required, opened a new log
I20251027 03:08:24.672986 30418 raft_consensus.cc:359] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ffdcbbac841b591e8b319f0b5dec5" member_type: VOTER }
I20251027 03:08:24.673028 30418 raft_consensus.cc:385] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:08:24.673051 30418 raft_consensus.cc:740] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642ffdcbbac841b591e8b319f0b5dec5, State: Initialized, Role: FOLLOWER
I20251027 03:08:24.673100 30418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ffdcbbac841b591e8b319f0b5dec5" member_type: VOTER }
I20251027 03:08:24.673143 30418 raft_consensus.cc:399] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:08:24.673166 30418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:08:24.673194 30418 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:08:24.673630 30418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ffdcbbac841b591e8b319f0b5dec5" member_type: VOTER }
I20251027 03:08:24.673682 30418 leader_election.cc:304] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 642ffdcbbac841b591e8b319f0b5dec5; no voters:
I20251027 03:08:24.673772 30418 leader_election.cc:290] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:08:24.673826 30421 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:08:24.673913 30418 sys_catalog.cc:565] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:08:24.673951 30421 raft_consensus.cc:697] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 1 LEADER]: Becoming Leader. State: Replica: 642ffdcbbac841b591e8b319f0b5dec5, State: Running, Role: LEADER
I20251027 03:08:24.674027 30421 consensus_queue.cc:237] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ffdcbbac841b591e8b319f0b5dec5" member_type: VOTER }
I20251027 03:08:24.674206 30422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "642ffdcbbac841b591e8b319f0b5dec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ffdcbbac841b591e8b319f0b5dec5" member_type: VOTER } }
I20251027 03:08:24.674216 30423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 642ffdcbbac841b591e8b319f0b5dec5. Latest consensus state: current_term: 1 leader_uuid: "642ffdcbbac841b591e8b319f0b5dec5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642ffdcbbac841b591e8b319f0b5dec5" member_type: VOTER } }
I20251027 03:08:24.674372 30423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [sys.catalog]: This master's current role is: LEADER
I20251027 03:08:24.674299 30422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [sys.catalog]: This master's current role is: LEADER
I20251027 03:08:24.674546 30429 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:08:24.674679 30429 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:08:24.675199 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:08:24.675199 30429 catalog_manager.cc:1357] Generated new cluster ID: aa1b30ec5e3a4952a92a9047ea0d0935
I20251027 03:08:24.675266 30429 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:08:24.695371 30429 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:08:24.695765 30429 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:08:24.703605 30429 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5: Generated new TSK 0
I20251027 03:08:24.703670 30429 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:08:24.706739 28253 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 3.1168e-05s
I20251027 03:08:36.781793 28253 master.cc:561] Master@127.27.151.126:37815 shutting down...
I20251027 03:08:36.784266 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:08:36.784348 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:36.784366 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 642ffdcbbac841b591e8b319f0b5dec5: stopping tablet replica
I20251027 03:08:36.795624 28253 master.cc:583] Master@127.27.151.126:37815 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12140 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20251027 03:08:36.798102 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:35755
I20251027 03:08:36.798223 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:08:36.799348 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:36.799422 30443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:36.799464 30440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:36.799429 30441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:36.799651 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:36.799687 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:36.799729 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534516799729 us; error 0 us; skew 500 ppm
I20251027 03:08:36.800297 28253 webserver.cc:492] Webserver started at http://127.27.151.126:37825/ using document root <none> and password file <none>
I20251027 03:08:36.800374 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:36.800422 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:36.800468 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:36.800740 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "79f66d4350194bf8b66563b14404d026"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.801595 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:08:36.802065 30448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.802210 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.802244 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "79f66d4350194bf8b66563b14404d026"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.802287 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:36.812920 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:36.813066 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:36.815294 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:35755
I20251027 03:08:36.815867 30500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:35755 every 8 connection(s)
I20251027 03:08:36.815971 30501 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:36.816888 30501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026: Bootstrap starting.
I20251027 03:08:36.817103 30501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026: Neither blocks nor log segments found. Creating new log.
I20251027 03:08:36.817502 30501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026: No bootstrap required, opened a new log
I20251027 03:08:36.817593 30501 raft_consensus.cc:359] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f66d4350194bf8b66563b14404d026" member_type: VOTER }
I20251027 03:08:36.817631 30501 raft_consensus.cc:385] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:08:36.817644 30501 raft_consensus.cc:740] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f66d4350194bf8b66563b14404d026, State: Initialized, Role: FOLLOWER
I20251027 03:08:36.817674 30501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f66d4350194bf8b66563b14404d026" member_type: VOTER }
I20251027 03:08:36.817732 30501 raft_consensus.cc:399] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:08:36.817772 30501 raft_consensus.cc:493] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:08:36.817791 30501 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:08:36.818207 30501 raft_consensus.cc:515] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f66d4350194bf8b66563b14404d026" member_type: VOTER }
I20251027 03:08:36.818254 30501 leader_election.cc:304] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79f66d4350194bf8b66563b14404d026; no voters:
I20251027 03:08:36.818333 30501 leader_election.cc:290] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:08:36.818403 30504 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:08:36.818478 30501 sys_catalog.cc:565] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:08:36.818532 30504 raft_consensus.cc:697] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 1 LEADER]: Becoming Leader. State: Replica: 79f66d4350194bf8b66563b14404d026, State: Running, Role: LEADER
I20251027 03:08:36.818588 30504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f66d4350194bf8b66563b14404d026" member_type: VOTER }
I20251027 03:08:36.818843 30505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79f66d4350194bf8b66563b14404d026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f66d4350194bf8b66563b14404d026" member_type: VOTER } }
I20251027 03:08:36.818975 30505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [sys.catalog]: This master's current role is: LEADER
I20251027 03:08:36.818853 30506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79f66d4350194bf8b66563b14404d026. Latest consensus state: current_term: 1 leader_uuid: "79f66d4350194bf8b66563b14404d026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f66d4350194bf8b66563b14404d026" member_type: VOTER } }
I20251027 03:08:36.819204 30512 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:08:36.819221 30506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [sys.catalog]: This master's current role is: LEADER
I20251027 03:08:36.819397 30512 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:08:36.819844 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:08:36.819921 30512 catalog_manager.cc:1357] Generated new cluster ID: 6dd9171038144710b54d6f816e08d89b
I20251027 03:08:36.819964 30512 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:08:36.827358 30512 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:08:36.827746 30512 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:08:36.832794 30512 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026: Generated new TSK 0
I20251027 03:08:36.832875 30512 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:08:36.835390 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:36.836413 30525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:36.836474 30522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:36.836542 30523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:36.836585 28253 server_base.cc:1047] running on GCE node
I20251027 03:08:36.836707 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:36.836742 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:36.836769 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534516836770 us; error 0 us; skew 500 ppm
I20251027 03:08:36.837263 28253 webserver.cc:492] Webserver started at http://127.27.151.65:33817/ using document root <none> and password file <none>
I20251027 03:08:36.837337 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:36.837383 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:36.837431 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:36.837644 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "27f70936edc14f40abc483fff36c6d32"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.838444 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:08:36.838841 30530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.838961 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.839001 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "27f70936edc14f40abc483fff36c6d32"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.839046 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:36.847251 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:36.847448 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:36.847571 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:36.847747 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:36.847775 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.847805 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:36.847829 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.850484 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:40093
I20251027 03:08:36.850535 30593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:40093 every 8 connection(s)
I20251027 03:08:36.850809 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:36.851807 30599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:36.851816 30598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:36.852025 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:36.852246 30601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:36.852372 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:36.852404 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:36.852416 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534516852417 us; error 0 us; skew 500 ppm
I20251027 03:08:36.852880 28253 webserver.cc:492] Webserver started at http://127.27.151.66:39923/ using document root <none> and password file <none>
I20251027 03:08:36.852949 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:36.852993 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:36.853036 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:36.853269 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "06f266af5f454204807fb7171891182d"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.854316 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20251027 03:08:36.854710 30606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.854867 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.854912 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "06f266af5f454204807fb7171891182d"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.854946 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:36.857478 30594 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35755
I20251027 03:08:36.857537 30594 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:36.857625 30594 heartbeater.cc:507] Master 127.27.151.126:35755 requested a full tablet report, sending...
I20251027 03:08:36.857851 30465 ts_manager.cc:194] Registered new tserver with Master: 27f70936edc14f40abc483fff36c6d32 (127.27.151.65:40093)
I20251027 03:08:36.858686 30465 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48670
I20251027 03:08:36.868256 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:36.868471 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:36.868597 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:36.868772 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:36.868798 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.868829 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:36.868855 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.871421 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:35429
I20251027 03:08:36.871452 30669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:35429 every 8 connection(s)
I20251027 03:08:36.871719 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:36.872690 30673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:36.872732 30674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:36.872793 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:36.872931 30676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:36.873129 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:36.873171 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:36.873189 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534516873189 us; error 0 us; skew 500 ppm
I20251027 03:08:36.873670 28253 webserver.cc:492] Webserver started at http://127.27.151.67:32957/ using document root <none> and password file <none>
I20251027 03:08:36.873744 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:36.873787 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:36.873832 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:36.874106 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "f2332ad8d56243b589233683f05f9406"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.874806 30670 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35755
I20251027 03:08:36.874863 30670 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:36.874969 30670 heartbeater.cc:507] Master 127.27.151.126:35755 requested a full tablet report, sending...
I20251027 03:08:36.875191 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:08:36.875274 30465 ts_manager.cc:194] Registered new tserver with Master: 06f266af5f454204807fb7171891182d (127.27.151.66:35429)
I20251027 03:08:36.875653 30465 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48672
I20251027 03:08:36.875671 30681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.875815 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.875866 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "f2332ad8d56243b589233683f05f9406"
format_stamp: "Formatted at 2025-10-27 03:08:36 on dist-test-slave-4cc8"
I20251027 03:08:36.875908 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:36.896242 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:36.896450 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:36.896595 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:36.896790 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:36.896819 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.896849 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:36.896891 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:36.899777 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:33281
I20251027 03:08:36.899840 30744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:33281 every 8 connection(s)
I20251027 03:08:36.902499 30745 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35755
I20251027 03:08:36.902554 30745 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:36.902678 30745 heartbeater.cc:507] Master 127.27.151.126:35755 requested a full tablet report, sending...
I20251027 03:08:36.902876 30465 ts_manager.cc:194] Registered new tserver with Master: f2332ad8d56243b589233683f05f9406 (127.27.151.67:33281)
I20251027 03:08:36.903199 30465 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48674
I20251027 03:08:36.903275 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003327159s
I20251027 03:08:37.859228 30594 heartbeater.cc:499] Master 127.27.151.126:35755 was elected leader, sending a full tablet report...
I20251027 03:08:37.876161 30670 heartbeater.cc:499] Master 127.27.151.126:35755 was elected leader, sending a full tablet report...
I20251027 03:08:37.903604 30745 heartbeater.cc:499] Master 127.27.151.126:35755 was elected leader, sending a full tablet report...
I20251027 03:08:38.819825 30519 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:40.820021 30519 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:42.820240 30519 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:44.820444 30519 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:46.820662 30519 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:48.820870 30519 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:48.978627 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:08:48.981117 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:48.992434 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:08:48.993201 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:08:48.995226 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:49.006453 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:08:49.007062 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:08:49.009045 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:08:49.020108 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:08:49.020747 28253 master.cc:561] Master@127.27.151.126:35755 shutting down...
I20251027 03:08:49.022742 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:08:49.022799 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:08:49.022827 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 79f66d4350194bf8b66563b14404d026: stopping tablet replica
I20251027 03:08:49.034168 28253 master.cc:583] Master@127.27.151.126:35755 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12239 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20251027 03:08:49.037304 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:41021
I20251027 03:08:49.037446 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:49.038712 30749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:49.038749 30750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.038846 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:49.038800 30752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.039012 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:49.039048 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:49.039062 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534529039061 us; error 0 us; skew 500 ppm
I20251027 03:08:49.039556 28253 webserver.cc:492] Webserver started at http://127.27.151.126:39255/ using document root <none> and password file <none>
I20251027 03:08:49.039623 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:49.039660 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:49.039717 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:49.039952 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "b2d8561d419a48709e890888efe2066f"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.040766 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:08:49.041193 30757 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.041294 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.041335 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "b2d8561d419a48709e890888efe2066f"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.041380 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:49.048719 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:49.048862 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:49.051040 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:41021
I20251027 03:08:49.052556 30809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:41021 every 8 connection(s)
I20251027 03:08:49.052645 30810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:49.053475 30810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f: Bootstrap starting.
I20251027 03:08:49.053727 30810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f: Neither blocks nor log segments found. Creating new log.
I20251027 03:08:49.054119 30810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f: No bootstrap required, opened a new log
I20251027 03:08:49.054225 30810 raft_consensus.cc:359] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8561d419a48709e890888efe2066f" member_type: VOTER }
I20251027 03:08:49.054265 30810 raft_consensus.cc:385] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:08:49.054278 30810 raft_consensus.cc:740] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d8561d419a48709e890888efe2066f, State: Initialized, Role: FOLLOWER
I20251027 03:08:49.054322 30810 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8561d419a48709e890888efe2066f" member_type: VOTER }
I20251027 03:08:49.054352 30810 raft_consensus.cc:399] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:08:49.054373 30810 raft_consensus.cc:493] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:08:49.054395 30810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:08:49.054800 30810 raft_consensus.cc:515] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8561d419a48709e890888efe2066f" member_type: VOTER }
I20251027 03:08:49.054847 30810 leader_election.cc:304] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d8561d419a48709e890888efe2066f; no voters:
I20251027 03:08:49.054937 30810 leader_election.cc:290] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:08:49.054977 30813 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:08:49.055048 30810 sys_catalog.cc:565] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:08:49.055085 30813 raft_consensus.cc:697] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 1 LEADER]: Becoming Leader. State: Replica: b2d8561d419a48709e890888efe2066f, State: Running, Role: LEADER
I20251027 03:08:49.055130 30813 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8561d419a48709e890888efe2066f" member_type: VOTER }
I20251027 03:08:49.055311 30814 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d8561d419a48709e890888efe2066f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8561d419a48709e890888efe2066f" member_type: VOTER } }
I20251027 03:08:49.055321 30815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d8561d419a48709e890888efe2066f. Latest consensus state: current_term: 1 leader_uuid: "b2d8561d419a48709e890888efe2066f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8561d419a48709e890888efe2066f" member_type: VOTER } }
I20251027 03:08:49.055408 30815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [sys.catalog]: This master's current role is: LEADER
I20251027 03:08:49.055532 30814 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [sys.catalog]: This master's current role is: LEADER
I20251027 03:08:49.055624 30818 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:08:49.055769 30818 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:08:49.056284 30818 catalog_manager.cc:1357] Generated new cluster ID: 13c4613b60f64a3aa24517c6343273f3
I20251027 03:08:49.056324 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:08:49.056331 30818 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:08:49.060770 30818 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:08:49.061131 30818 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:08:49.065101 30818 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f: Generated new TSK 0
I20251027 03:08:49.065150 30818 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:08:49.071858 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:49.072961 30831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.073028 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:49.073083 30832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:49.073186 30834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.073288 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:49.073321 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:49.073356 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534529073357 us; error 0 us; skew 500 ppm
I20251027 03:08:49.073807 28253 webserver.cc:492] Webserver started at http://127.27.151.65:41587/ using document root <none> and password file <none>
I20251027 03:08:49.073870 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:49.073901 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:49.073940 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:49.074131 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "32cb354dc2a94b11afb422acddc26082"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.074914 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:08:49.075317 30839 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.075426 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.075469 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "32cb354dc2a94b11afb422acddc26082"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.075500 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:49.103901 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:49.104065 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:49.104202 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:49.104360 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:49.104383 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.104406 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:49.104421 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.106875 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:42311
I20251027 03:08:49.106916 30902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:42311 every 8 connection(s)
I20251027 03:08:49.107209 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:49.108261 30907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:49.108353 30910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.108420 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:49.108465 30908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.108616 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:49.108654 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:49.108670 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534529108671 us; error 0 us; skew 500 ppm
I20251027 03:08:49.109189 28253 webserver.cc:492] Webserver started at http://127.27.151.66:37395/ using document root <none> and password file <none>
I20251027 03:08:49.109265 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:49.109311 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:49.109357 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:49.109618 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "a08a2df4498b432792b94a1f9513bf63"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.110471 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:08:49.110847 30915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.110967 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.111016 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "a08a2df4498b432792b94a1f9513bf63"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.111053 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:49.114372 30903 heartbeater.cc:344] Connected to a master server at 127.27.151.126:41021
I20251027 03:08:49.114424 30903 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:49.114516 30903 heartbeater.cc:507] Master 127.27.151.126:41021 requested a full tablet report, sending...
I20251027 03:08:49.114729 30774 ts_manager.cc:194] Registered new tserver with Master: 32cb354dc2a94b11afb422acddc26082 (127.27.151.65:42311)
I20251027 03:08:49.115229 30774 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41578
I20251027 03:08:49.116726 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:49.116885 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:49.117002 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:49.117156 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:49.117182 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.117210 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:49.117235 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.119944 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:45899
I20251027 03:08:49.119974 30978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:45899 every 8 connection(s)
I20251027 03:08:49.120263 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:49.121315 30983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.121654 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:49.121361 30982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:49.121543 30985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.122603 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:49.122651 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:49.122671 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534529122671 us; error 0 us; skew 500 ppm
I20251027 03:08:49.122977 30979 heartbeater.cc:344] Connected to a master server at 127.27.151.126:41021
I20251027 03:08:49.123034 30979 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:49.123113 30979 heartbeater.cc:507] Master 127.27.151.126:41021 requested a full tablet report, sending...
I20251027 03:08:49.123131 28253 webserver.cc:492] Webserver started at http://127.27.151.67:35295/ using document root <none> and password file <none>
I20251027 03:08:49.123186 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:49.123261 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:49.123306 30774 ts_manager.cc:194] Registered new tserver with Master: a08a2df4498b432792b94a1f9513bf63 (127.27.151.66:45899)
I20251027 03:08:49.123310 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:49.123553 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "a50f9830f6bc4de69ae1a6812da52ba7"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.123628 30774 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41584
I20251027 03:08:49.124406 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:08:49.124783 30990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.124871 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.124909 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "a50f9830f6bc4de69ae1a6812da52ba7"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.124958 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:49.137615 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:49.137815 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:49.137954 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:49.138126 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:49.138152 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.138180 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:49.138206 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.141225 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:39547
I20251027 03:08:49.141270 31053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:39547 every 8 connection(s)
I20251027 03:08:49.141508 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:08:49.142562 31057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:08:49.142676 31060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.142887 28253 server_base.cc:1047] running on GCE node
W20251027 03:08:49.143709 31058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:08:49.143916 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:08:49.143965 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:08:49.143981 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534529143982 us; error 0 us; skew 500 ppm
I20251027 03:08:49.144184 31054 heartbeater.cc:344] Connected to a master server at 127.27.151.126:41021
I20251027 03:08:49.144229 31054 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:49.144310 31054 heartbeater.cc:507] Master 127.27.151.126:41021 requested a full tablet report, sending...
I20251027 03:08:49.144510 30774 ts_manager.cc:194] Registered new tserver with Master: a50f9830f6bc4de69ae1a6812da52ba7 (127.27.151.67:39547)
I20251027 03:08:49.144574 28253 webserver.cc:492] Webserver started at http://127.27.151.68:43691/ using document root <none> and password file <none>
I20251027 03:08:49.144645 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:08:49.144692 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:08:49.144737 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:08:49.144945 30774 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41586
I20251027 03:08:49.144987 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "02ae06d392a149b28e30f298036c41cf"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.145895 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:08:49.146248 31065 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.146358 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.146404 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "02ae06d392a149b28e30f298036c41cf"
format_stamp: "Formatted at 2025-10-27 03:08:49 on dist-test-slave-4cc8"
I20251027 03:08:49.146438 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:08:49.159679 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:08:49.159854 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:08:49.159960 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:08:49.160084 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:08:49.160113 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.160135 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:08:49.160181 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:49.163252 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:45677
I20251027 03:08:49.163302 31128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:45677 every 8 connection(s)
I20251027 03:08:49.165858 31129 heartbeater.cc:344] Connected to a master server at 127.27.151.126:41021
I20251027 03:08:49.165902 31129 heartbeater.cc:461] Registering TS with master...
I20251027 03:08:49.165973 31129 heartbeater.cc:507] Master 127.27.151.126:41021 requested a full tablet report, sending...
I20251027 03:08:49.166137 30774 ts_manager.cc:194] Registered new tserver with Master: 02ae06d392a149b28e30f298036c41cf (127.27.151.68:45677)
I20251027 03:08:49.166388 30774 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41592
I20251027 03:08:49.166798 28253 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.003304068s
I20251027 03:08:50.115726 30903 heartbeater.cc:499] Master 127.27.151.126:41021 was elected leader, sending a full tablet report...
I20251027 03:08:50.124115 30979 heartbeater.cc:499] Master 127.27.151.126:41021 was elected leader, sending a full tablet report...
I20251027 03:08:50.145552 31054 heartbeater.cc:499] Master 127.27.151.126:41021 was elected leader, sending a full tablet report...
I20251027 03:08:50.166817 31129 heartbeater.cc:499] Master 127.27.151.126:41021 was elected leader, sending a full tablet report...
I20251027 03:08:50.194460 28253 test_util.cc:276] Using random seed: 938380314
I20251027 03:08:50.198402 30774 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41614:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 03:08:50.202682 31093 tablet_service.cc:1505] Processing CreateTablet for tablet e18e766d665c44519f967c60bc6dcb63 (DEFAULT_TABLE table=test-workload [id=0176a58d2de64330ac53b3f596b2f2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:08:50.202801 31093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e766d665c44519f967c60bc6dcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:50.203081 30943 tablet_service.cc:1505] Processing CreateTablet for tablet e18e766d665c44519f967c60bc6dcb63 (DEFAULT_TABLE table=test-workload [id=0176a58d2de64330ac53b3f596b2f2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:08:50.203224 30943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e766d665c44519f967c60bc6dcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:50.203671 31018 tablet_service.cc:1505] Processing CreateTablet for tablet e18e766d665c44519f967c60bc6dcb63 (DEFAULT_TABLE table=test-workload [id=0176a58d2de64330ac53b3f596b2f2b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:08:50.203781 31018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e766d665c44519f967c60bc6dcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:08:50.204190 31150 tablet_bootstrap.cc:492] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf: Bootstrap starting.
I20251027 03:08:50.204497 31151 tablet_bootstrap.cc:492] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: Bootstrap starting.
I20251027 03:08:50.204591 31150 tablet_bootstrap.cc:654] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf: Neither blocks nor log segments found. Creating new log.
I20251027 03:08:50.204952 31152 tablet_bootstrap.cc:492] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7: Bootstrap starting.
I20251027 03:08:50.205008 31151 tablet_bootstrap.cc:654] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: Neither blocks nor log segments found. Creating new log.
I20251027 03:08:50.205369 31152 tablet_bootstrap.cc:654] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7: Neither blocks nor log segments found. Creating new log.
I20251027 03:08:50.205577 31150 tablet_bootstrap.cc:492] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf: No bootstrap required, opened a new log
I20251027 03:08:50.205618 31150 ts_tablet_manager.cc:1403] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:08:50.205751 31150 raft_consensus.cc:359] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.205823 31150 raft_consensus.cc:385] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:08:50.205843 31150 raft_consensus.cc:740] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ae06d392a149b28e30f298036c41cf, State: Initialized, Role: FOLLOWER
I20251027 03:08:50.205904 31150 consensus_queue.cc:260] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.205925 31152 tablet_bootstrap.cc:492] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7: No bootstrap required, opened a new log
I20251027 03:08:50.205984 31152 ts_tablet_manager.cc:1403] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:08:50.206053 31150 ts_tablet_manager.cc:1434] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:08:50.206127 31152 raft_consensus.cc:359] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.206197 31152 raft_consensus.cc:385] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:08:50.206233 31152 raft_consensus.cc:740] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50f9830f6bc4de69ae1a6812da52ba7, State: Initialized, Role: FOLLOWER
I20251027 03:08:50.206291 31152 consensus_queue.cc:260] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.206329 31151 tablet_bootstrap.cc:492] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: No bootstrap required, opened a new log
I20251027 03:08:50.206362 31151 ts_tablet_manager.cc:1403] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:08:50.206430 31152 ts_tablet_manager.cc:1434] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:50.206498 31151 raft_consensus.cc:359] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.206535 31151 raft_consensus.cc:385] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:08:50.206547 31151 raft_consensus.cc:740] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08a2df4498b432792b94a1f9513bf63, State: Initialized, Role: FOLLOWER
I20251027 03:08:50.206581 31151 consensus_queue.cc:260] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.206705 31151 ts_tablet_manager.cc:1434] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:08:50.209008 31156 raft_consensus.cc:493] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:08:50.209056 31156 raft_consensus.cc:515] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.209196 31156 leader_election.cc:290] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a08a2df4498b432792b94a1f9513bf63 (127.27.151.66:45899), a50f9830f6bc4de69ae1a6812da52ba7 (127.27.151.67:39547)
I20251027 03:08:50.211639 30953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18e766d665c44519f967c60bc6dcb63" candidate_uuid: "02ae06d392a149b28e30f298036c41cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a08a2df4498b432792b94a1f9513bf63" is_pre_election: true
I20251027 03:08:50.211733 30953 raft_consensus.cc:2468] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ae06d392a149b28e30f298036c41cf in term 0.
I20251027 03:08:50.211869 31066 leader_election.cc:304] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ae06d392a149b28e30f298036c41cf, a08a2df4498b432792b94a1f9513bf63; no voters:
I20251027 03:08:50.211942 31156 raft_consensus.cc:2804] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:08:50.211911 31028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18e766d665c44519f967c60bc6dcb63" candidate_uuid: "02ae06d392a149b28e30f298036c41cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" is_pre_election: true
I20251027 03:08:50.211982 31156 raft_consensus.cc:493] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:08:50.211997 31028 raft_consensus.cc:2468] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ae06d392a149b28e30f298036c41cf in term 0.
I20251027 03:08:50.212007 31156 raft_consensus.cc:3060] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:08:50.212551 31156 raft_consensus.cc:515] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.212646 31156 leader_election.cc:290] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [CANDIDATE]: Term 1 election: Requested vote from peers a08a2df4498b432792b94a1f9513bf63 (127.27.151.66:45899), a50f9830f6bc4de69ae1a6812da52ba7 (127.27.151.67:39547)
I20251027 03:08:50.212785 31028 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18e766d665c44519f967c60bc6dcb63" candidate_uuid: "02ae06d392a149b28e30f298036c41cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a50f9830f6bc4de69ae1a6812da52ba7"
I20251027 03:08:50.212798 30953 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e18e766d665c44519f967c60bc6dcb63" candidate_uuid: "02ae06d392a149b28e30f298036c41cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a08a2df4498b432792b94a1f9513bf63"
I20251027 03:08:50.212852 30953 raft_consensus.cc:3060] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:08:50.212842 31028 raft_consensus.cc:3060] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:08:50.213407 31028 raft_consensus.cc:2468] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ae06d392a149b28e30f298036c41cf in term 1.
I20251027 03:08:50.213404 30953 raft_consensus.cc:2468] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ae06d392a149b28e30f298036c41cf in term 1.
I20251027 03:08:50.213546 31066 leader_election.cc:304] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ae06d392a149b28e30f298036c41cf, a08a2df4498b432792b94a1f9513bf63; no voters:
I20251027 03:08:50.213618 31156 raft_consensus.cc:2804] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:08:50.213742 31156 raft_consensus.cc:697] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 1 LEADER]: Becoming Leader. State: Replica: 02ae06d392a149b28e30f298036c41cf, State: Running, Role: LEADER
I20251027 03:08:50.213786 31156 consensus_queue.cc:237] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } }
I20251027 03:08:50.214205 30773 catalog_manager.cc:5649] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ae06d392a149b28e30f298036c41cf (127.27.151.68). New cstate: current_term: 1 leader_uuid: "02ae06d392a149b28e30f298036c41cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02ae06d392a149b28e30f298036c41cf" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 } health_report { overall_health: UNKNOWN } } }
W20251027 03:08:50.370787 30980 tablet.cc:2378] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 03:08:50.626955 31156 consensus_queue.cc:1048] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08a2df4498b432792b94a1f9513bf63" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:08:50.628454 31166 consensus_queue.cc:1048] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a50f9830f6bc4de69ae1a6812da52ba7" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 39547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:08:51.059705 30829 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:51.059801 30829 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:51.059836 30829 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:53.059947 30829 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:53.060040 30829 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:53.060058 30829 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:55.060189 30829 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:55.060287 30829 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:55.060304 30829 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:57.060420 30829 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:57.060516 30829 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:57.060534 30829 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:08:59.060740 30829 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:08:59.060885 30829 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:08:59.060918 30829 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:01.061055 30829 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:01.061172 30829 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:01.061192 30829 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:01.266618 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:09:01.269343 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:01.281261 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:09:01.282076 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:09:01.284590 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:01.284718 28253 tablet_replica.cc:333] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63: stopping tablet replica
I20251027 03:09:01.284771 28253 raft_consensus.cc:2243] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:01.284822 28253 raft_consensus.cc:2272] T e18e766d665c44519f967c60bc6dcb63 P a08a2df4498b432792b94a1f9513bf63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:01.296201 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:09:01.296921 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:09:01.299279 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:01.299424 28253 tablet_replica.cc:333] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7: stopping tablet replica
I20251027 03:09:01.299485 28253 raft_consensus.cc:2243] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:01.299535 28253 raft_consensus.cc:2272] T e18e766d665c44519f967c60bc6dcb63 P a50f9830f6bc4de69ae1a6812da52ba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:01.311523 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:09:01.312318 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:09:01.315202 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:01.315305 28253 tablet_replica.cc:333] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf: stopping tablet replica
I20251027 03:09:01.315354 28253 raft_consensus.cc:2243] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:01.315418 28253 raft_consensus.cc:2272] T e18e766d665c44519f967c60bc6dcb63 P 02ae06d392a149b28e30f298036c41cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:01.326829 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:09:01.327548 28253 master.cc:561] Master@127.27.151.126:41021 shutting down...
I20251027 03:09:01.329784 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:01.329850 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:01.329890 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P b2d8561d419a48709e890888efe2066f: stopping tablet replica
I20251027 03:09:01.341241 28253 master.cc:583] Master@127.27.151.126:41021 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12308 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20251027 03:09:01.345419 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:32957
I20251027 03:09:01.345553 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:01.346621 31170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:01.346752 31173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.346796 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:01.346884 31171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.346989 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:01.347023 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:01.347043 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534541347042 us; error 0 us; skew 500 ppm
I20251027 03:09:01.347561 28253 webserver.cc:492] Webserver started at http://127.27.151.126:39265/ using document root <none> and password file <none>
I20251027 03:09:01.347645 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:01.347693 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:01.347738 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:01.347957 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "615a61d0753d4d77a50034adf7f9d209"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.348796 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:01.349180 31178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.349298 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.349336 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "615a61d0753d4d77a50034adf7f9d209"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.349380 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:01.356163 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:01.356309 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:01.358412 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:32957
I20251027 03:09:01.360396 31230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:32957 every 8 connection(s)
I20251027 03:09:01.360502 31231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:01.361366 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209: Bootstrap starting.
I20251027 03:09:01.361579 31231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:01.361989 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209: No bootstrap required, opened a new log
I20251027 03:09:01.362087 31231 raft_consensus.cc:359] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a61d0753d4d77a50034adf7f9d209" member_type: VOTER }
I20251027 03:09:01.362126 31231 raft_consensus.cc:385] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:01.362139 31231 raft_consensus.cc:740] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615a61d0753d4d77a50034adf7f9d209, State: Initialized, Role: FOLLOWER
I20251027 03:09:01.362177 31231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a61d0753d4d77a50034adf7f9d209" member_type: VOTER }
I20251027 03:09:01.362221 31231 raft_consensus.cc:399] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:09:01.362239 31231 raft_consensus.cc:493] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:09:01.362277 31231 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:01.362689 31231 raft_consensus.cc:515] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a61d0753d4d77a50034adf7f9d209" member_type: VOTER }
I20251027 03:09:01.362738 31231 leader_election.cc:304] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615a61d0753d4d77a50034adf7f9d209; no voters:
I20251027 03:09:01.362823 31231 leader_election.cc:290] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:09:01.362890 31234 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:01.363001 31231 sys_catalog.cc:565] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:09:01.363010 31234 raft_consensus.cc:697] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 1 LEADER]: Becoming Leader. State: Replica: 615a61d0753d4d77a50034adf7f9d209, State: Running, Role: LEADER
I20251027 03:09:01.363123 31234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a61d0753d4d77a50034adf7f9d209" member_type: VOTER }
I20251027 03:09:01.363343 31235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615a61d0753d4d77a50034adf7f9d209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a61d0753d4d77a50034adf7f9d209" member_type: VOTER } }
I20251027 03:09:01.363474 31235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:01.363372 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615a61d0753d4d77a50034adf7f9d209. Latest consensus state: current_term: 1 leader_uuid: "615a61d0753d4d77a50034adf7f9d209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615a61d0753d4d77a50034adf7f9d209" member_type: VOTER } }
I20251027 03:09:01.363643 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:01.363662 31240 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:09:01.363806 31240 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:09:01.364212 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:09:01.364611 31240 catalog_manager.cc:1357] Generated new cluster ID: 516cd151bddc46269d77d0b32e60f818
I20251027 03:09:01.364653 31240 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:09:01.371269 31240 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:09:01.371620 31240 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:09:01.381052 31240 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209: Generated new TSK 0
I20251027 03:09:01.381119 31240 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:09:01.395993 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:01.397363 31253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.397408 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:01.397379 31255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:01.397396 31252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.397614 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:01.397650 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:01.397665 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534541397666 us; error 0 us; skew 500 ppm
I20251027 03:09:01.398177 28253 webserver.cc:492] Webserver started at http://127.27.151.65:37223/ using document root <none> and password file <none>
I20251027 03:09:01.398242 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:01.398273 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:01.398308 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:01.398514 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "5382e62b772f4a70917e6d3c5271daa9"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.399292 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:01.399699 31260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.399824 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.399875 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "5382e62b772f4a70917e6d3c5271daa9"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.399910 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:01.409219 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:01.409364 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:01.409466 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:01.409597 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:01.409619 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.409644 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:01.409667 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.412196 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:41235
I20251027 03:09:01.412222 31323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:41235 every 8 connection(s)
I20251027 03:09:01.412516 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:01.413479 31328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:01.413532 31329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.413926 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:01.414386 31331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.414526 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:01.414569 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:01.414582 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534541414582 us; error 0 us; skew 500 ppm
I20251027 03:09:01.415131 28253 webserver.cc:492] Webserver started at http://127.27.151.66:43083/ using document root <none> and password file <none>
I20251027 03:09:01.415220 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:01.415264 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:01.415297 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:01.415535 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "9779bd4961a142af87a81701b5385a46"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.415719 31324 heartbeater.cc:344] Connected to a master server at 127.27.151.126:32957
I20251027 03:09:01.415776 31324 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:01.415865 31324 heartbeater.cc:507] Master 127.27.151.126:32957 requested a full tablet report, sending...
I20251027 03:09:01.416075 31195 ts_manager.cc:194] Registered new tserver with Master: 5382e62b772f4a70917e6d3c5271daa9 (127.27.151.65:41235)
I20251027 03:09:01.416558 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:01.416672 31195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40132
I20251027 03:09:01.417062 31336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.417194 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.417238 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "9779bd4961a142af87a81701b5385a46"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.417282 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:01.425627 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:01.425837 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:01.425974 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:01.426141 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:01.426167 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.426198 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:01.426224 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.428896 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:33237
I20251027 03:09:01.428951 31399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:33237 every 8 connection(s)
I20251027 03:09:01.429242 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:01.430438 31403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:01.430449 31406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.430651 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:01.431234 31404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:01.431406 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:01.431455 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:01.431479 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534541431479 us; error 0 us; skew 500 ppm
I20251027 03:09:01.432063 28253 webserver.cc:492] Webserver started at http://127.27.151.67:33833/ using document root <none> and password file <none>
I20251027 03:09:01.432117 31400 heartbeater.cc:344] Connected to a master server at 127.27.151.126:32957
I20251027 03:09:01.432158 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:01.432188 31400 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:01.432202 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:01.432238 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:01.432263 31400 heartbeater.cc:507] Master 127.27.151.126:32957 requested a full tablet report, sending...
I20251027 03:09:01.432467 31195 ts_manager.cc:194] Registered new tserver with Master: 9779bd4961a142af87a81701b5385a46 (127.27.151.66:33237)
I20251027 03:09:01.432474 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "a24fcce9a61546a7afeb029b951592bb"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.432976 31195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40136
I20251027 03:09:01.433461 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:01.433890 31411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.434018 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.434058 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "a24fcce9a61546a7afeb029b951592bb"
format_stamp: "Formatted at 2025-10-27 03:09:01 on dist-test-slave-4cc8"
I20251027 03:09:01.434096 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:01.452993 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:01.453228 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:01.453372 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:01.453543 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:01.453570 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.453589 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:01.453619 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:01.456681 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:34679
I20251027 03:09:01.456714 31474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:34679 every 8 connection(s)
I20251027 03:09:01.459363 31475 heartbeater.cc:344] Connected to a master server at 127.27.151.126:32957
I20251027 03:09:01.459420 31475 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:01.459519 31475 heartbeater.cc:507] Master 127.27.151.126:32957 requested a full tablet report, sending...
I20251027 03:09:01.459733 31195 ts_manager.cc:194] Registered new tserver with Master: a24fcce9a61546a7afeb029b951592bb (127.27.151.67:34679)
I20251027 03:09:01.460047 31195 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40142
I20251027 03:09:01.460125 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003311855s
I20251027 03:09:02.417112 31324 heartbeater.cc:499] Master 127.27.151.126:32957 was elected leader, sending a full tablet report...
I20251027 03:09:02.433429 31400 heartbeater.cc:499] Master 127.27.151.126:32957 was elected leader, sending a full tablet report...
I20251027 03:09:02.460599 31475 heartbeater.cc:499] Master 127.27.151.126:32957 was elected leader, sending a full tablet report...
I20251027 03:09:02.487782 28253 test_util.cc:276] Using random seed: 950673636
I20251027 03:09:02.492201 31195 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40156:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:09:02.492326 31195 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:09:02.496843 31439 tablet_service.cc:1505] Processing CreateTablet for tablet bc23368e3f8f49df8ea525e5a024dbcc (DEFAULT_TABLE table=test-workload [id=6ffafca42b5e460da0b2d8291e8f7af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:09:02.496981 31439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc23368e3f8f49df8ea525e5a024dbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:02.497373 31364 tablet_service.cc:1505] Processing CreateTablet for tablet bc23368e3f8f49df8ea525e5a024dbcc (DEFAULT_TABLE table=test-workload [id=6ffafca42b5e460da0b2d8291e8f7af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:09:02.497465 31288 tablet_service.cc:1505] Processing CreateTablet for tablet bc23368e3f8f49df8ea525e5a024dbcc (DEFAULT_TABLE table=test-workload [id=6ffafca42b5e460da0b2d8291e8f7af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:09:02.497498 31364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc23368e3f8f49df8ea525e5a024dbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:02.497567 31288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc23368e3f8f49df8ea525e5a024dbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:02.498580 31497 tablet_bootstrap.cc:492] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb: Bootstrap starting.
I20251027 03:09:02.498987 31497 tablet_bootstrap.cc:654] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:02.499017 31498 tablet_bootstrap.cc:492] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9: Bootstrap starting.
I20251027 03:09:02.499382 31496 tablet_bootstrap.cc:492] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46: Bootstrap starting.
I20251027 03:09:02.499469 31498 tablet_bootstrap.cc:654] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:02.499583 31497 tablet_bootstrap.cc:492] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb: No bootstrap required, opened a new log
I20251027 03:09:02.499632 31497 ts_tablet_manager.cc:1403] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:02.499807 31497 raft_consensus.cc:359] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.499887 31497 raft_consensus.cc:385] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:02.499930 31497 raft_consensus.cc:740] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24fcce9a61546a7afeb029b951592bb, State: Initialized, Role: FOLLOWER
I20251027 03:09:02.499982 31497 consensus_queue.cc:260] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.500135 31498 tablet_bootstrap.cc:492] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9: No bootstrap required, opened a new log
I20251027 03:09:02.500180 31497 ts_tablet_manager.cc:1434] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:02.500208 31498 ts_tablet_manager.cc:1403] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:02.500387 31498 raft_consensus.cc:359] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.500445 31498 raft_consensus.cc:385] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:02.500469 31498 raft_consensus.cc:740] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5382e62b772f4a70917e6d3c5271daa9, State: Initialized, Role: FOLLOWER
I20251027 03:09:02.500552 31498 consensus_queue.cc:260] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.500605 31496 tablet_bootstrap.cc:654] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:02.501062 31498 ts_tablet_manager.cc:1434] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:02.501453 31496 tablet_bootstrap.cc:492] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46: No bootstrap required, opened a new log
I20251027 03:09:02.501521 31496 ts_tablet_manager.cc:1403] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:09:02.501634 31496 raft_consensus.cc:359] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.501678 31496 raft_consensus.cc:385] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:02.501691 31496 raft_consensus.cc:740] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9779bd4961a142af87a81701b5385a46, State: Initialized, Role: FOLLOWER
I20251027 03:09:02.501732 31496 consensus_queue.cc:260] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.501859 31496 ts_tablet_manager.cc:1434] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:09:02.691866 31502 raft_consensus.cc:493] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:02.691985 31502 raft_consensus.cc:515] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.692175 31502 leader_election.cc:290] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a24fcce9a61546a7afeb029b951592bb (127.27.151.67:34679), 9779bd4961a142af87a81701b5385a46 (127.27.151.66:33237)
I20251027 03:09:02.694762 31449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc23368e3f8f49df8ea525e5a024dbcc" candidate_uuid: "5382e62b772f4a70917e6d3c5271daa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24fcce9a61546a7afeb029b951592bb" is_pre_election: true
I20251027 03:09:02.694808 31374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc23368e3f8f49df8ea525e5a024dbcc" candidate_uuid: "5382e62b772f4a70917e6d3c5271daa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9779bd4961a142af87a81701b5385a46" is_pre_election: true
I20251027 03:09:02.694846 31449 raft_consensus.cc:2468] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5382e62b772f4a70917e6d3c5271daa9 in term 0.
I20251027 03:09:02.694868 31374 raft_consensus.cc:2468] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5382e62b772f4a70917e6d3c5271daa9 in term 0.
I20251027 03:09:02.695005 31261 leader_election.cc:304] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5382e62b772f4a70917e6d3c5271daa9, 9779bd4961a142af87a81701b5385a46; no voters:
I20251027 03:09:02.695086 31502 raft_consensus.cc:2804] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:02.695124 31502 raft_consensus.cc:493] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:02.695142 31502 raft_consensus.cc:3060] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:02.695694 31502 raft_consensus.cc:515] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.695781 31502 leader_election.cc:290] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [CANDIDATE]: Term 1 election: Requested vote from peers a24fcce9a61546a7afeb029b951592bb (127.27.151.67:34679), 9779bd4961a142af87a81701b5385a46 (127.27.151.66:33237)
I20251027 03:09:02.695971 31374 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc23368e3f8f49df8ea525e5a024dbcc" candidate_uuid: "5382e62b772f4a70917e6d3c5271daa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9779bd4961a142af87a81701b5385a46"
I20251027 03:09:02.696017 31374 raft_consensus.cc:3060] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:02.696203 31449 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc23368e3f8f49df8ea525e5a024dbcc" candidate_uuid: "5382e62b772f4a70917e6d3c5271daa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a24fcce9a61546a7afeb029b951592bb"
I20251027 03:09:02.696256 31449 raft_consensus.cc:3060] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:02.696589 31374 raft_consensus.cc:2468] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5382e62b772f4a70917e6d3c5271daa9 in term 1.
I20251027 03:09:02.696751 31261 leader_election.cc:304] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5382e62b772f4a70917e6d3c5271daa9, 9779bd4961a142af87a81701b5385a46; no voters:
I20251027 03:09:02.696833 31502 raft_consensus.cc:2804] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:02.696918 31449 raft_consensus.cc:2468] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5382e62b772f4a70917e6d3c5271daa9 in term 1.
I20251027 03:09:02.696990 31502 raft_consensus.cc:697] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 1 LEADER]: Becoming Leader. State: Replica: 5382e62b772f4a70917e6d3c5271daa9, State: Running, Role: LEADER
I20251027 03:09:02.697052 31502 consensus_queue.cc:237] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } }
I20251027 03:09:02.697592 31194 catalog_manager.cc:5649] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5382e62b772f4a70917e6d3c5271daa9 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "5382e62b772f4a70917e6d3c5271daa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5382e62b772f4a70917e6d3c5271daa9" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41235 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:03.103538 31502 consensus_queue.cc:1048] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9779bd4961a142af87a81701b5385a46" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:03.105089 31507 consensus_queue.cc:1048] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a24fcce9a61546a7afeb029b951592bb" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 34679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:03.364250 31249 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:03.364383 31249 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:03.364418 31249 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:03.364579 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:04.364993 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:05.364547 31249 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:05.364673 31249 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:05.364693 31249 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:05.365299 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:06.365731 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:07.364830 31249 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:07.364967 31249 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:07.364991 31249 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:07.366041 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:08.366406 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:09.365135 31249 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:09.365259 31249 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:09.365278 31249 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:09.366721 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:10.367017 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:11.365414 31249 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:11.365542 31249 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:11.365584 31249 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:11.367333 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:12.367676 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:13.365732 31249 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:13.365865 31249 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:13.365901 31249 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:13.367998 31244 placement_policy_util.cc:407] tablet bc23368e3f8f49df8ea525e5a024dbcc: detected majority of replicas (2 of 3) at location L0
I20251027 03:09:13.769379 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:09:13.772063 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:13.772184 28253 tablet_replica.cc:333] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9: stopping tablet replica
I20251027 03:09:13.772248 28253 raft_consensus.cc:2243] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:13.772311 28253 raft_consensus.cc:2272] T bc23368e3f8f49df8ea525e5a024dbcc P 5382e62b772f4a70917e6d3c5271daa9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:13.784634 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:09:13.785454 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:09:13.788098 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:13.788242 28253 tablet_replica.cc:333] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46: stopping tablet replica
I20251027 03:09:13.788292 28253 raft_consensus.cc:2243] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:13.788336 28253 raft_consensus.cc:2272] T bc23368e3f8f49df8ea525e5a024dbcc P 9779bd4961a142af87a81701b5385a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:13.799587 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:09:13.800273 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:09:13.803050 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:13.803155 28253 tablet_replica.cc:333] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb: stopping tablet replica
I20251027 03:09:13.803215 28253 raft_consensus.cc:2243] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:13.803262 28253 raft_consensus.cc:2272] T bc23368e3f8f49df8ea525e5a024dbcc P a24fcce9a61546a7afeb029b951592bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:13.814512 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:09:13.815192 28253 master.cc:561] Master@127.27.151.126:32957 shutting down...
I20251027 03:09:13.817510 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:13.817564 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:13.817593 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 615a61d0753d4d77a50034adf7f9d209: stopping tablet replica
I20251027 03:09:13.828809 28253 master.cc:583] Master@127.27.151.126:32957 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12487 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20251027 03:09:13.832984 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:37795
I20251027 03:09:13.833119 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:09:13.834215 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:13.834270 31521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:13.834331 31518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:13.834340 31519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:13.834544 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:13.834574 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:13.834585 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534553834586 us; error 0 us; skew 500 ppm
I20251027 03:09:13.835029 28253 webserver.cc:492] Webserver started at http://127.27.151.126:44963/ using document root <none> and password file <none>
I20251027 03:09:13.835101 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:13.835151 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:13.835186 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:13.835397 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "c6144ee9e79f41fc948fc9c34a5a33e0"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.836231 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:13.836607 31526 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.836726 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.836764 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "c6144ee9e79f41fc948fc9c34a5a33e0"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.836807 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:13.851161 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:13.851315 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:13.853613 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:37795
I20251027 03:09:13.855217 31578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:37795 every 8 connection(s)
I20251027 03:09:13.855319 31579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:13.856199 31579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0: Bootstrap starting.
I20251027 03:09:13.856416 31579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:13.856889 31579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0: No bootstrap required, opened a new log
I20251027 03:09:13.856988 31579 raft_consensus.cc:359] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" member_type: VOTER }
I20251027 03:09:13.857026 31579 raft_consensus.cc:385] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:13.857039 31579 raft_consensus.cc:740] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6144ee9e79f41fc948fc9c34a5a33e0, State: Initialized, Role: FOLLOWER
I20251027 03:09:13.857071 31579 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" member_type: VOTER }
I20251027 03:09:13.857105 31579 raft_consensus.cc:399] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:09:13.857124 31579 raft_consensus.cc:493] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:09:13.857147 31579 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:13.857548 31579 raft_consensus.cc:515] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" member_type: VOTER }
I20251027 03:09:13.857599 31579 leader_election.cc:304] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6144ee9e79f41fc948fc9c34a5a33e0; no voters:
I20251027 03:09:13.857693 31579 leader_election.cc:290] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:09:13.857759 31582 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:13.857833 31579 sys_catalog.cc:565] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:09:13.857972 31582 raft_consensus.cc:697] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 1 LEADER]: Becoming Leader. State: Replica: c6144ee9e79f41fc948fc9c34a5a33e0, State: Running, Role: LEADER
I20251027 03:09:13.858032 31582 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" member_type: VOTER }
I20251027 03:09:13.858273 31583 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" member_type: VOTER } }
I20251027 03:09:13.858301 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6144ee9e79f41fc948fc9c34a5a33e0. Latest consensus state: current_term: 1 leader_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6144ee9e79f41fc948fc9c34a5a33e0" member_type: VOTER } }
I20251027 03:09:13.858347 31583 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:13.858366 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:13.858543 31588 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:09:13.858728 31588 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:09:13.859277 31588 catalog_manager.cc:1357] Generated new cluster ID: 3360a1be96a64103b19d0125a6b6d4a8
I20251027 03:09:13.859282 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:09:13.859333 31588 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:09:13.873883 31588 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:09:13.874282 31588 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:09:13.884824 31588 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0: Generated new TSK 0
I20251027 03:09:13.884923 31588 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:09:13.890939 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:13.892035 31600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:13.892220 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:13.892266 31603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:13.892364 31601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:13.892532 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:13.892571 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:13.892592 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534553892593 us; error 0 us; skew 500 ppm
I20251027 03:09:13.893121 28253 webserver.cc:492] Webserver started at http://127.27.151.65:39345/ using document root <none> and password file <none>
I20251027 03:09:13.893189 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:13.893222 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:13.893256 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:13.893462 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "78e34a95a4a44fde91fa13a71fa3da26"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.894277 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:13.894745 31608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.894879 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.894929 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "78e34a95a4a44fde91fa13a71fa3da26"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.894963 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:13.907126 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:13.907275 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:13.907369 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:13.907490 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:13.907510 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.907533 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:13.907549 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.910004 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:32889
I20251027 03:09:13.910032 31671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:32889 every 8 connection(s)
I20251027 03:09:13.910300 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:13.911170 31676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:13.911237 31677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:13.911432 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:13.911430 31679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:13.911576 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:13.911603 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:13.911624 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534553911623 us; error 0 us; skew 500 ppm
I20251027 03:09:13.912184 28253 webserver.cc:492] Webserver started at http://127.27.151.66:44059/ using document root <none> and password file <none>
I20251027 03:09:13.912264 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:13.912303 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:13.912353 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:13.912571 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "50aa97f11c164e0b9e3520267cf6c85f"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.913504 31672 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37795
I20251027 03:09:13.913518 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:13.913571 31672 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:13.913682 31672 heartbeater.cc:507] Master 127.27.151.126:37795 requested a full tablet report, sending...
I20251027 03:09:13.913877 31543 ts_manager.cc:194] Registered new tserver with Master: 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:13.914014 31684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.914126 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.914167 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "50aa97f11c164e0b9e3520267cf6c85f"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.914206 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:13.914561 31543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48448
I20251027 03:09:13.927284 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:13.927441 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:13.927551 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:13.927702 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:13.927726 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.927754 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:13.927780 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.930372 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:41091
I20251027 03:09:13.930426 31747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:41091 every 8 connection(s)
I20251027 03:09:13.930688 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:13.931550 31751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:13.931581 31752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:13.931746 31754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:13.931766 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:13.931944 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:13.931977 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:13.931999 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534553931999 us; error 0 us; skew 500 ppm
I20251027 03:09:13.932449 28253 webserver.cc:492] Webserver started at http://127.27.151.67:38607/ using document root <none> and password file <none>
I20251027 03:09:13.932518 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:13.932560 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:13.932603 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:13.932853 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "10153e76038242fdaec0d401cdbb0411"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.933589 31748 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37795
I20251027 03:09:13.933634 31748 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:13.933749 31748 heartbeater.cc:507] Master 127.27.151.126:37795 requested a full tablet report, sending...
I20251027 03:09:13.933791 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:13.933971 31543 ts_manager.cc:194] Registered new tserver with Master: 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:13.934260 31759 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.934363 31543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48460
I20251027 03:09:13.934378 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.934437 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "10153e76038242fdaec0d401cdbb0411"
format_stamp: "Formatted at 2025-10-27 03:09:13 on dist-test-slave-4cc8"
I20251027 03:09:13.934485 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:13.947772 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:13.947955 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:13.948084 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:13.948268 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:13.948293 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.948324 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:13.948352 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:13.951411 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:37045
I20251027 03:09:13.951462 31822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:37045 every 8 connection(s)
I20251027 03:09:13.954133 31823 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37795
I20251027 03:09:13.954181 31823 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:13.954257 31823 heartbeater.cc:507] Master 127.27.151.126:37795 requested a full tablet report, sending...
I20251027 03:09:13.954525 31543 ts_manager.cc:194] Registered new tserver with Master: 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:13.954882 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003326113s
I20251027 03:09:13.954986 31543 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48464
I20251027 03:09:14.915056 31672 heartbeater.cc:499] Master 127.27.151.126:37795 was elected leader, sending a full tablet report...
I20251027 03:09:14.934799 31748 heartbeater.cc:499] Master 127.27.151.126:37795 was elected leader, sending a full tablet report...
I20251027 03:09:14.955525 31823 heartbeater.cc:499] Master 127.27.151.126:37795 was elected leader, sending a full tablet report...
I20251027 03:09:14.979928 28253 test_util.cc:276] Using random seed: 963165781
I20251027 03:09:14.985034 31543 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48480:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:09:14.985189 31543 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:09:14.991591 31636 tablet_service.cc:1505] Processing CreateTablet for tablet 11ce7fc7b4334ffe99df2944dca2af4a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251027 03:09:14.991763 31636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ce7fc7b4334ffe99df2944dca2af4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.991812 31634 tablet_service.cc:1505] Processing CreateTablet for tablet 0e2464cf0e9243b7845c98d8cde8528b (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251027 03:09:14.991923 31634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2464cf0e9243b7845c98d8cde8528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.992911 31636 tablet_service.cc:1505] Processing CreateTablet for tablet 30ef3ff09aa247fab2eae47a3f1718fd (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251027 03:09:14.992928 31843 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:14.992962 31634 tablet_service.cc:1505] Processing CreateTablet for tablet 42bb6d5c4d0e42af9c337417582580b6 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251027 03:09:14.992990 31636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ef3ff09aa247fab2eae47a3f1718fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.993095 31633 tablet_service.cc:1505] Processing CreateTablet for tablet 645f3c62551e42109ebfe785bde36416 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251027 03:09:14.993124 31634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bb6d5c4d0e42af9c337417582580b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.993171 31633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645f3c62551e42109ebfe785bde36416. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.993348 31843 tablet_bootstrap.cc:654] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:14.994216 31632 tablet_service.cc:1505] Processing CreateTablet for tablet 5e1ec5a8f7224329be0fe0e5453d71fc (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251027 03:09:14.994287 31632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e1ec5a8f7224329be0fe0e5453d71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.995021 31631 tablet_service.cc:1505] Processing CreateTablet for tablet 726dd98dfd6c47a5b95b723de71cb5c5 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251027 03:09:14.995110 31631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726dd98dfd6c47a5b95b723de71cb5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.995868 31630 tablet_service.cc:1505] Processing CreateTablet for tablet bb4aebbafbe6415e90eaf6f6feb6b760 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251027 03:09:14.995950 31630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4aebbafbe6415e90eaf6f6feb6b760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.997213 31629 tablet_service.cc:1505] Processing CreateTablet for tablet 914edf6a5ed5402fb1d07b6afba645b6 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251027 03:09:14.997313 31629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914edf6a5ed5402fb1d07b6afba645b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.998489 31628 tablet_service.cc:1505] Processing CreateTablet for tablet c176d9f872b84ac986522990949395eb (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251027 03:09:14.998574 31628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176d9f872b84ac986522990949395eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.999668 31627 tablet_service.cc:1505] Processing CreateTablet for tablet a7a4ca641cf349f29eeb0292a996449a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251027 03:09:14.999753 31627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a4ca641cf349f29eeb0292a996449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:14.991770 31635 tablet_service.cc:1505] Processing CreateTablet for tablet 6045341ee60c472bbb01e952650a735a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251027 03:09:15.001094 31635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6045341ee60c472bbb01e952650a735a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.002619 31712 tablet_service.cc:1505] Processing CreateTablet for tablet 11ce7fc7b4334ffe99df2944dca2af4a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251027 03:09:15.002707 31712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ce7fc7b4334ffe99df2944dca2af4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.004034 31712 tablet_service.cc:1505] Processing CreateTablet for tablet 30ef3ff09aa247fab2eae47a3f1718fd (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251027 03:09:15.004113 31712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ef3ff09aa247fab2eae47a3f1718fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.004240 31845 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.004680 31845 tablet_bootstrap.cc:654] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.005204 31712 tablet_service.cc:1505] Processing CreateTablet for tablet 42bb6d5c4d0e42af9c337417582580b6 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251027 03:09:15.005300 31712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bb6d5c4d0e42af9c337417582580b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.006340 31711 tablet_service.cc:1505] Processing CreateTablet for tablet 6045341ee60c472bbb01e952650a735a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251027 03:09:15.006412 31711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6045341ee60c472bbb01e952650a735a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.007071 31845 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.007159 31845 ts_tablet_manager.cc:1403] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:09:15.007302 31845 raft_consensus.cc:359] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.007378 31845 raft_consensus.cc:385] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.007401 31845 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.007464 31845 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.007539 31710 tablet_service.cc:1505] Processing CreateTablet for tablet 0e2464cf0e9243b7845c98d8cde8528b (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251027 03:09:15.007637 31710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2464cf0e9243b7845c98d8cde8528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.007665 31845 ts_tablet_manager.cc:1434] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.007794 31845 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.008376 31845 tablet_bootstrap.cc:654] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.008890 31709 tablet_service.cc:1505] Processing CreateTablet for tablet 645f3c62551e42109ebfe785bde36416 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251027 03:09:15.008965 31845 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.008984 31709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645f3c62551e42109ebfe785bde36416. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.009066 31708 tablet_service.cc:1505] Processing CreateTablet for tablet 5e1ec5a8f7224329be0fe0e5453d71fc (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251027 03:09:15.009150 31708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e1ec5a8f7224329be0fe0e5453d71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.009971 31707 tablet_service.cc:1505] Processing CreateTablet for tablet 726dd98dfd6c47a5b95b723de71cb5c5 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251027 03:09:15.010051 31707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726dd98dfd6c47a5b95b723de71cb5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.010135 31845 ts_tablet_manager.cc:1403] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:15.010308 31845 raft_consensus.cc:359] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.010378 31845 raft_consensus.cc:385] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.010442 31845 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.010492 31845 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.010648 31845 ts_tablet_manager.cc:1434] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.010776 31845 tablet_bootstrap.cc:492] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.011124 31706 tablet_service.cc:1505] Processing CreateTablet for tablet bb4aebbafbe6415e90eaf6f6feb6b760 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251027 03:09:15.011215 31706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4aebbafbe6415e90eaf6f6feb6b760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.011220 31845 tablet_bootstrap.cc:654] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.011880 31845 tablet_bootstrap.cc:492] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.011924 31845 ts_tablet_manager.cc:1403] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.012064 31845 raft_consensus.cc:359] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.012121 31845 raft_consensus.cc:385] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.012169 31845 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.012238 31705 tablet_service.cc:1505] Processing CreateTablet for tablet 914edf6a5ed5402fb1d07b6afba645b6 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251027 03:09:15.012285 31703 tablet_service.cc:1505] Processing CreateTablet for tablet a7a4ca641cf349f29eeb0292a996449a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251027 03:09:15.012341 31705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914edf6a5ed5402fb1d07b6afba645b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.012432 31703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a4ca641cf349f29eeb0292a996449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.012257 31704 tablet_service.cc:1505] Processing CreateTablet for tablet c176d9f872b84ac986522990949395eb (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251027 03:09:15.013934 31784 tablet_service.cc:1505] Processing CreateTablet for tablet 645f3c62551e42109ebfe785bde36416 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251027 03:09:15.013949 31704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176d9f872b84ac986522990949395eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.014040 31787 tablet_service.cc:1505] Processing CreateTablet for tablet 11ce7fc7b4334ffe99df2944dca2af4a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251027 03:09:15.014060 31784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645f3c62551e42109ebfe785bde36416. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.014130 31787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ce7fc7b4334ffe99df2944dca2af4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.012233 31845 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.015385 31787 tablet_service.cc:1505] Processing CreateTablet for tablet 30ef3ff09aa247fab2eae47a3f1718fd (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251027 03:09:15.015403 31784 tablet_service.cc:1505] Processing CreateTablet for tablet 42bb6d5c4d0e42af9c337417582580b6 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251027 03:09:15.015400 31845 ts_tablet_manager.cc:1434] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251027 03:09:15.015465 31787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ef3ff09aa247fab2eae47a3f1718fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.015478 31845 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.015622 31784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bb6d5c4d0e42af9c337417582580b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.015792 31845 tablet_bootstrap.cc:654] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.016287 31778 tablet_service.cc:1505] Processing CreateTablet for tablet a7a4ca641cf349f29eeb0292a996449a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251027 03:09:15.016369 31778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a4ca641cf349f29eeb0292a996449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.016486 31851 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.016870 31851 tablet_bootstrap.cc:654] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.017274 31786 tablet_service.cc:1505] Processing CreateTablet for tablet 6045341ee60c472bbb01e952650a735a (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251027 03:09:15.017335 31786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6045341ee60c472bbb01e952650a735a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.018043 31785 tablet_service.cc:1505] Processing CreateTablet for tablet 0e2464cf0e9243b7845c98d8cde8528b (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251027 03:09:15.018097 31785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2464cf0e9243b7845c98d8cde8528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.018708 31783 tablet_service.cc:1505] Processing CreateTablet for tablet 5e1ec5a8f7224329be0fe0e5453d71fc (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251027 03:09:15.018760 31783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e1ec5a8f7224329be0fe0e5453d71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.019397 31782 tablet_service.cc:1505] Processing CreateTablet for tablet 726dd98dfd6c47a5b95b723de71cb5c5 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251027 03:09:15.019448 31782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726dd98dfd6c47a5b95b723de71cb5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.020179 31781 tablet_service.cc:1505] Processing CreateTablet for tablet bb4aebbafbe6415e90eaf6f6feb6b760 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251027 03:09:15.020246 31781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4aebbafbe6415e90eaf6f6feb6b760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.020964 31780 tablet_service.cc:1505] Processing CreateTablet for tablet 914edf6a5ed5402fb1d07b6afba645b6 (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251027 03:09:15.021019 31780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914edf6a5ed5402fb1d07b6afba645b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.021668 31779 tablet_service.cc:1505] Processing CreateTablet for tablet c176d9f872b84ac986522990949395eb (DEFAULT_TABLE table=test-workload [id=dee44af73a894cc0aa4f501bdf139afe]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251027 03:09:15.021721 31779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176d9f872b84ac986522990949395eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:15.022759 31845 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.022809 31845 ts_tablet_manager.cc:1403] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20251027 03:09:15.022926 31845 raft_consensus.cc:359] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.022976 31851 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.023069 31851 ts_tablet_manager.cc:1403] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20251027 03:09:15.023216 31851 raft_consensus.cc:359] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.023298 31843 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.023324 31851 raft_consensus.cc:385] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.023363 31843 ts_tablet_manager.cc:1403] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.000s
I20251027 03:09:15.022975 31845 raft_consensus.cc:385] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.023496 31845 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.023440 31851 raft_consensus.cc:740] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.023605 31845 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.023586 31843 raft_consensus.cc:359] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.023728 31843 raft_consensus.cc:385] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.023782 31843 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.023830 31843 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.023742 31851 consensus_queue.cc:260] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.023983 31843 ts_tablet_manager.cc:1434] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.024034 31843 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.024106 31851 ts_tablet_manager.cc:1434] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:15.024255 31851 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.024426 31843 tablet_bootstrap.cc:654] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.024713 31851 tablet_bootstrap.cc:654] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.025342 31843 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.025391 31843 ts_tablet_manager.cc:1403] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.025532 31843 raft_consensus.cc:359] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.025578 31843 raft_consensus.cc:385] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.025591 31843 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.025625 31843 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.025715 31843 ts_tablet_manager.cc:1434] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.025767 31843 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.026075 31843 tablet_bootstrap.cc:654] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.023741 31845 ts_tablet_manager.cc:1434] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.026561 31843 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.026645 31843 ts_tablet_manager.cc:1403] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.026808 31843 raft_consensus.cc:359] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.026860 31843 raft_consensus.cc:385] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.026880 31843 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.026923 31843 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.026961 31845 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.026996 31843 ts_tablet_manager.cc:1434] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.027045 31843 tablet_bootstrap.cc:492] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.027438 31843 tablet_bootstrap.cc:654] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.027659 31845 tablet_bootstrap.cc:654] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.028053 31843 tablet_bootstrap.cc:492] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.028081 31851 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.028108 31843 ts_tablet_manager.cc:1403] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.028311 31845 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.028357 31845 ts_tablet_manager.cc:1403] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.028316 31843 raft_consensus.cc:359] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.028455 31843 raft_consensus.cc:385] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.028122 31851 ts_tablet_manager.cc:1403] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251027 03:09:15.028520 31845 raft_consensus.cc:359] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.028635 31845 raft_consensus.cc:385] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.028512 31843 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.028690 31851 raft_consensus.cc:359] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.028698 31845 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.028781 31843 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.028836 31845 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.028939 31843 ts_tablet_manager.cc:1434] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.029001 31843 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.029006 31845 ts_tablet_manager.cc:1434] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.029100 31845 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.029419 31843 tablet_bootstrap.cc:654] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.029464 31845 tablet_bootstrap.cc:654] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.028784 31851 raft_consensus.cc:385] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.029764 31851 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.029841 31851 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.030009 31851 ts_tablet_manager.cc:1434] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.030174 31851 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.030611 31851 tablet_bootstrap.cc:654] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.030659 31843 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.030704 31843 ts_tablet_manager.cc:1403] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:15.030835 31843 raft_consensus.cc:359] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.030880 31843 raft_consensus.cc:385] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.030911 31843 raft_consensus.cc:740] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.030954 31843 consensus_queue.cc:260] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.031165 31845 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.031215 31851 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.031271 31851 ts_tablet_manager.cc:1403] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:15.031370 31843 ts_tablet_manager.cc:1434] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.031406 31851 raft_consensus.cc:359] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.031466 31851 raft_consensus.cc:385] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.031489 31851 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.031538 31851 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.031617 31851 ts_tablet_manager.cc:1434] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.031683 31851 tablet_bootstrap.cc:492] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.031695 31843 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.032032 31843 tablet_bootstrap.cc:654] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.032078 31851 tablet_bootstrap.cc:654] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.031210 31845 ts_tablet_manager.cc:1403] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:15.032593 31845 raft_consensus.cc:359] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.032624 31851 tablet_bootstrap.cc:492] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.032648 31845 raft_consensus.cc:385] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.032671 31851 ts_tablet_manager.cc:1403] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.032672 31845 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.032775 31845 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.032873 31845 ts_tablet_manager.cc:1434] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.032856 31851 raft_consensus.cc:359] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.032930 31851 raft_consensus.cc:385] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.032958 31851 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.033010 31851 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.033113 31851 ts_tablet_manager.cc:1434] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:09:15.033115 31845 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.033174 31851 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.033596 31851 tablet_bootstrap.cc:654] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.033679 31845 tablet_bootstrap.cc:654] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.034111 31843 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.034160 31843 ts_tablet_manager.cc:1403] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:15.034256 31851 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.034298 31851 ts_tablet_manager.cc:1403] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.034310 31843 raft_consensus.cc:359] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.034374 31843 raft_consensus.cc:385] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.034396 31843 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.034457 31843 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.034502 31851 raft_consensus.cc:359] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.034663 31851 raft_consensus.cc:385] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.034693 31851 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.034548 31843 ts_tablet_manager.cc:1434] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.034741 31851 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.034845 31851 ts_tablet_manager.cc:1434] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.034905 31851 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.034871 31843 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.035327 31851 tablet_bootstrap.cc:654] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.035413 31843 tablet_bootstrap.cc:654] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.035899 31851 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.035964 31851 ts_tablet_manager.cc:1403] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.036015 31843 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.036062 31843 ts_tablet_manager.cc:1403] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.036103 31851 raft_consensus.cc:359] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.036223 31851 raft_consensus.cc:385] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.036211 31843 raft_consensus.cc:359] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.036264 31851 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.036280 31843 raft_consensus.cc:385] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.036307 31843 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.036324 31851 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.036342 31843 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.036410 31843 ts_tablet_manager.cc:1434] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.036459 31843 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.036532 31851 ts_tablet_manager.cc:1434] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.035759 31845 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.036581 31845 ts_tablet_manager.cc:1403] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251027 03:09:15.036725 31845 raft_consensus.cc:359] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.036768 31845 raft_consensus.cc:385] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.036790 31845 raft_consensus.cc:740] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.036844 31845 consensus_queue.cc:260] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.037061 31845 ts_tablet_manager.cc:1434] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.037129 31845 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.036863 31843 tablet_bootstrap.cc:654] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.037465 31854 raft_consensus.cc:493] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.037501 31854 raft_consensus.cc:515] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.037670 31854 leader_election.cc:290] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.038108 31851 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.038239 31845 tablet_bootstrap.cc:654] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.038443 31843 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.038527 31843 ts_tablet_manager.cc:1403] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:09:15.038580 31851 tablet_bootstrap.cc:654] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.038784 31843 raft_consensus.cc:359] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.038909 31843 raft_consensus.cc:385] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.038980 31843 raft_consensus.cc:740] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.038834 31845 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.039088 31845 ts_tablet_manager.cc:1403] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:15.039114 31843 consensus_queue.cc:260] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.039287 31851 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.039291 31843 ts_tablet_manager.cc:1434] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.039350 31851 ts_tablet_manager.cc:1403] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.039515 31843 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.039565 31851 raft_consensus.cc:359] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.039630 31851 raft_consensus.cc:385] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.039665 31851 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.039706 31851 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.039793 31851 ts_tablet_manager.cc:1434] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.039850 31851 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.040130 31843 tablet_bootstrap.cc:654] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.040293 31851 tablet_bootstrap.cc:654] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.039224 31845 raft_consensus.cc:359] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.040485 31845 raft_consensus.cc:385] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.040518 31845 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.040562 31845 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.040855 31851 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.040905 31851 ts_tablet_manager.cc:1403] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:15.040963 31845 ts_tablet_manager.cc:1434] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 03:09:15.041018 31843 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.041062 31843 ts_tablet_manager.cc:1403] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:15.041056 31851 raft_consensus.cc:359] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.041122 31851 raft_consensus.cc:385] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.041149 31851 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.041213 31851 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.041213 31843 raft_consensus.cc:359] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.041133 31845 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.041334 31851 ts_tablet_manager.cc:1434] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.041857 31851 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.042258 31851 tablet_bootstrap.cc:654] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.042783 31845 tablet_bootstrap.cc:654] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.043164 31851 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.043207 31851 ts_tablet_manager.cc:1403] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.043344 31851 raft_consensus.cc:359] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.043390 31851 raft_consensus.cc:385] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.043411 31851 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.043459 31851 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.043869 31851 ts_tablet_manager.cc:1434] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.043953 31851 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.043977 31845 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.044018 31845 ts_tablet_manager.cc:1403] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20251027 03:09:15.044179 31845 raft_consensus.cc:359] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.044239 31845 raft_consensus.cc:385] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.044263 31845 raft_consensus.cc:740] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.044304 31845 consensus_queue.cc:260] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.044315 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.044373 31722 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.044391 31845 ts_tablet_manager.cc:1434] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.044466 31845 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.044548 31851 tablet_bootstrap.cc:654] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.044541 31611 leader_election.cc:304] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.044668 31854 raft_consensus.cc:2804] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.044705 31854 raft_consensus.cc:493] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.044732 31854 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.044868 31845 tablet_bootstrap.cc:654] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.041409 31843 raft_consensus.cc:385] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.045192 31843 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.045240 31843 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.045387 31843 ts_tablet_manager.cc:1434] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20251027 03:09:15.045414 31854 raft_consensus.cc:515] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.045549 31854 leader_election.cc:290] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.045569 31845 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.045454 31843 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.045703 31845 ts_tablet_manager.cc:1403] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:15.045771 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.045845 31722 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.045974 31845 raft_consensus.cc:359] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.046070 31845 raft_consensus.cc:385] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.046082 31843 tablet_bootstrap.cc:654] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.046435 31845 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.046517 31722 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.046538 31845 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.046624 31843 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.046742 31843 ts_tablet_manager.cc:1403] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.046813 31845 ts_tablet_manager.cc:1434] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.046819 31611 leader_election.cc:304] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.046908 31843 raft_consensus.cc:359] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.046952 31854 raft_consensus.cc:2804] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.046977 31843 raft_consensus.cc:385] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.047003 31843 raft_consensus.cc:740] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.047060 31854 raft_consensus.cc:697] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:15.046690 31851 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.047149 31851 ts_tablet_manager.cc:1403] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:09:15.047049 31843 consensus_queue.cc:260] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.047317 31851 raft_consensus.cc:359] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.047379 31851 raft_consensus.cc:385] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.047112 31854 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.047408 31851 raft_consensus.cc:740] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.047451 31851 consensus_queue.cc:260] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.047526 31851 ts_tablet_manager.cc:1434] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.047585 31851 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.047919 31851 tablet_bootstrap.cc:654] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.048184 31797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.048200 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.048273 31797 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.047343 31843 ts_tablet_manager.cc:1434] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.048395 31851 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.048427 31851 ts_tablet_manager.cc:1403] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:15.048436 31843 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.048537 31851 raft_consensus.cc:359] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.048590 31851 raft_consensus.cc:385] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.048611 31851 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.048653 31851 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.048739 31851 ts_tablet_manager.cc:1434] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.048784 31843 tablet_bootstrap.cc:654] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.048789 31851 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: Bootstrap starting.
I20251027 03:09:15.049114 31542 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.049333 31843 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.049371 31843 ts_tablet_manager.cc:1403] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.049475 31843 raft_consensus.cc:359] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.049532 31843 raft_consensus.cc:385] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.049551 31843 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.049602 31843 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.049710 31843 ts_tablet_manager.cc:1434] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.049757 31851 tablet_bootstrap.cc:654] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.049762 31843 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: Bootstrap starting.
I20251027 03:09:15.046911 31845 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.050246 31851 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: No bootstrap required, opened a new log
I20251027 03:09:15.050266 31843 tablet_bootstrap.cc:654] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.050292 31851 ts_tablet_manager.cc:1403] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:09:15.050415 31851 raft_consensus.cc:359] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.050465 31851 raft_consensus.cc:385] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.050493 31851 raft_consensus.cc:740] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.050541 31851 consensus_queue.cc:260] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.050634 31851 ts_tablet_manager.cc:1434] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.050812 31843 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: No bootstrap required, opened a new log
I20251027 03:09:15.050850 31843 ts_tablet_manager.cc:1403] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.050983 31845 tablet_bootstrap.cc:654] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.050984 31843 raft_consensus.cc:359] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.051075 31843 raft_consensus.cc:385] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.051097 31843 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.051146 31843 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.051465 31843 ts_tablet_manager.cc:1434] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:09:15.051518 31845 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.051569 31845 ts_tablet_manager.cc:1403] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251027 03:09:15.051695 31845 raft_consensus.cc:359] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.051756 31845 raft_consensus.cc:385] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.051774 31845 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.051831 31845 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.051934 31845 ts_tablet_manager.cc:1434] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.051987 31845 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: Bootstrap starting.
I20251027 03:09:15.052556 31845 tablet_bootstrap.cc:654] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:15.053154 31845 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: No bootstrap required, opened a new log
I20251027 03:09:15.053195 31845 ts_tablet_manager.cc:1403] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:15.053299 31845 raft_consensus.cc:359] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.053339 31845 raft_consensus.cc:385] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:15.053351 31845 raft_consensus.cc:740] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Initialized, Role: FOLLOWER
I20251027 03:09:15.053402 31845 consensus_queue.cc:260] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.053504 31845 ts_tablet_manager.cc:1434] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.080672 31847 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.080760 31847 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.080875 31847 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.083434 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.083464 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.083518 31646 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.083537 31796 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.083690 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.083804 31847 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.083853 31847 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.083874 31847 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.084466 31847 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.084568 31847 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.084736 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.084740 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.084820 31646 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.084832 31796 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.085547 31796 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.085711 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.085744 31646 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.085827 31847 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.085911 31847 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:15.085976 31847 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.086486 31542 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.114317 31855 raft_consensus.cc:493] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.114373 31855 raft_consensus.cc:515] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.114522 31855 leader_election.cc:290] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.116961 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.117053 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.117054 31722 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 0.
I20251027 03:09:15.117144 31646 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 0.
I20251027 03:09:15.117264 31762 leader_election.cc:304] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.117340 31855 raft_consensus.cc:2804] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.117381 31855 raft_consensus.cc:493] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.117396 31855 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.117870 31855 raft_consensus.cc:515] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.117977 31855 leader_election.cc:290] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.118129 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.118175 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.118188 31646 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.118261 31722 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.118709 31646 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 1.
I20251027 03:09:15.118868 31760 leader_election.cc:304] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.118901 31722 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 1.
I20251027 03:09:15.118952 31855 raft_consensus.cc:2804] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.119068 31855 raft_consensus.cc:697] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:15.119122 31855 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.119612 31542 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:15.192018 31883 raft_consensus.cc:493] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.192106 31883 raft_consensus.cc:515] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.192281 31883 leader_election.cc:290] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.192425 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.192425 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.192531 31796 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.192535 31722 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.192667 31611 leader_election.cc:304] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.192771 31883 raft_consensus.cc:2804] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.192814 31883 raft_consensus.cc:493] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.192843 31883 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.193465 31883 raft_consensus.cc:515] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.193563 31883 leader_election.cc:290] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.193732 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.193739 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.193812 31722 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.193818 31796 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.194419 31796 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.194419 31722 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.194588 31611 leader_election.cc:304] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.194675 31883 raft_consensus.cc:2804] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.194715 31883 raft_consensus.cc:697] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:15.194761 31883 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.195441 31542 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.199189 31847 raft_consensus.cc:493] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.199236 31847 raft_consensus.cc:515] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.199352 31847 leader_election.cc:290] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.199532 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.199580 31646 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.199549 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.199638 31796 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.199702 31685 leader_election.cc:304] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.199803 31847 raft_consensus.cc:2804] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.199844 31847 raft_consensus.cc:493] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.199879 31847 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.200373 31847 raft_consensus.cc:515] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.200465 31847 leader_election.cc:290] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.200582 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.200639 31646 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.200695 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.200920 31796 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.201133 31646 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.201282 31685 leader_election.cc:304] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.201355 31847 raft_consensus.cc:2804] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.201417 31847 raft_consensus.cc:697] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:15.201427 31796 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.201479 31847 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.202036 31542 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.209311 31849 raft_consensus.cc:493] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.209360 31849 raft_consensus.cc:515] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.209453 31849 leader_election.cc:290] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:15.209600 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.209609 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.209669 31796 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.209686 31646 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.209831 31685 leader_election.cc:304] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.209908 31849 raft_consensus.cc:2804] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.209941 31849 raft_consensus.cc:493] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.209962 31849 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.210402 31849 raft_consensus.cc:515] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.210482 31849 leader_election.cc:290] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:15.210609 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.210625 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.210682 31796 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.210686 31646 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.211079 31796 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.211133 31646 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.211256 31685 leader_election.cc:304] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.211337 31849 raft_consensus.cc:2804] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.211395 31849 raft_consensus.cc:697] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:15.211441 31849 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.211921 31542 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:15.244166 31883 raft_consensus.cc:493] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.244240 31883 raft_consensus.cc:515] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.244413 31883 leader_election.cc:290] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.244599 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.244601 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.244699 31796 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.244709 31722 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.244834 31611 leader_election.cc:304] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.244946 31883 raft_consensus.cc:2804] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.244998 31883 raft_consensus.cc:493] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.245018 31883 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.245566 31883 raft_consensus.cc:515] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.245694 31883 leader_election.cc:290] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.245826 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.245834 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.245889 31722 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.245932 31796 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.246382 31796 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.246390 31722 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.246541 31611 leader_election.cc:304] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.246654 31883 raft_consensus.cc:2804] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.246718 31883 raft_consensus.cc:697] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:15.246764 31883 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.247304 31542 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.249373 31855 raft_consensus.cc:493] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.249420 31855 raft_consensus.cc:515] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.249532 31855 leader_election.cc:290] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.249648 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.249707 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.249711 31722 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 0.
I20251027 03:09:15.249780 31646 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 0.
I20251027 03:09:15.249881 31762 leader_election.cc:304] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.250002 31855 raft_consensus.cc:2804] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.250036 31855 raft_consensus.cc:493] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.250057 31855 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.250551 31855 raft_consensus.cc:515] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.250677 31855 leader_election.cc:290] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.250787 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.250793 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.250841 31722 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.250854 31646 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.251286 31722 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 1.
I20251027 03:09:15.251405 31646 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 1.
I20251027 03:09:15.251447 31762 leader_election.cc:304] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.251533 31855 raft_consensus.cc:2804] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.251585 31855 raft_consensus.cc:697] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:15.251638 31855 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.252101 31542 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:15.270884 31849 raft_consensus.cc:493] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.270943 31849 raft_consensus.cc:515] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.271063 31849 leader_election.cc:290] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.271226 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176d9f872b84ac986522990949395eb" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.271226 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176d9f872b84ac986522990949395eb" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.271325 31796 raft_consensus.cc:2468] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.271329 31646 raft_consensus.cc:2468] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.271473 31685 leader_election.cc:304] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.271551 31849 raft_consensus.cc:2804] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.271592 31849 raft_consensus.cc:493] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.271620 31849 raft_consensus.cc:3060] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.272074 31849 raft_consensus.cc:515] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.272194 31849 leader_election.cc:290] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.272331 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176d9f872b84ac986522990949395eb" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.272330 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c176d9f872b84ac986522990949395eb" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.272394 31646 raft_consensus.cc:3060] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.272405 31796 raft_consensus.cc:3060] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.272820 31646 raft_consensus.cc:2468] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.272823 31796 raft_consensus.cc:2468] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.272969 31685 leader_election.cc:304] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.273042 31849 raft_consensus.cc:2804] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.273097 31849 raft_consensus.cc:697] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:15.273133 31849 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.273633 31542 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.286281 31849 raft_consensus.cc:493] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.286330 31849 raft_consensus.cc:515] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.286435 31849 leader_election.cc:290] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.286566 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb4aebbafbe6415e90eaf6f6feb6b760" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.286583 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb4aebbafbe6415e90eaf6f6feb6b760" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.286618 31646 raft_consensus.cc:2468] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.286631 31796 raft_consensus.cc:2468] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 0.
I20251027 03:09:15.286741 31685 leader_election.cc:304] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.286800 31849 raft_consensus.cc:2804] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.286842 31849 raft_consensus.cc:493] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.286864 31849 raft_consensus.cc:3060] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.287272 31849 raft_consensus.cc:515] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.287365 31849 leader_election.cc:290] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.287511 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb4aebbafbe6415e90eaf6f6feb6b760" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.287524 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb4aebbafbe6415e90eaf6f6feb6b760" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.287567 31646 raft_consensus.cc:3060] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.287580 31796 raft_consensus.cc:3060] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.288002 31646 raft_consensus.cc:2468] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.288058 31796 raft_consensus.cc:2468] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 1.
I20251027 03:09:15.288187 31685 leader_election.cc:304] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.288281 31849 raft_consensus.cc:2804] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.288341 31849 raft_consensus.cc:697] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:15.288403 31849 consensus_queue.cc:237] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.288918 31542 catalog_manager.cc:5649] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 0 to 1, leader changed from <none> to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.327927 31883 raft_consensus.cc:493] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.327999 31883 raft_consensus.cc:515] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.328128 31883 leader_election.cc:290] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.328346 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645f3c62551e42109ebfe785bde36416" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.328325 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645f3c62551e42109ebfe785bde36416" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411" is_pre_election: true
I20251027 03:09:15.328425 31722 raft_consensus.cc:2468] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.328444 31796 raft_consensus.cc:2468] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 0.
I20251027 03:09:15.328827 31609 leader_election.cc:304] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.328887 31855 raft_consensus.cc:493] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:15.328934 31855 raft_consensus.cc:515] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.328936 31883 raft_consensus.cc:2804] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.329006 31883 raft_consensus.cc:493] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.329036 31883 raft_consensus.cc:3060] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.329038 31855 leader_election.cc:290] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.329152 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f" is_pre_election: true
I20251027 03:09:15.329180 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26" is_pre_election: true
I20251027 03:09:15.329201 31722 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 0.
I20251027 03:09:15.329242 31646 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 0.
I20251027 03:09:15.329322 31762 leader_election.cc:304] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.329391 31855 raft_consensus.cc:2804] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:15.329419 31855 raft_consensus.cc:493] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:15.329437 31855 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.329722 31883 raft_consensus.cc:515] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.329840 31883 leader_election.cc:290] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:15.329939 31722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645f3c62551e42109ebfe785bde36416" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.329983 31722 raft_consensus.cc:3060] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.330010 31796 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645f3c62551e42109ebfe785bde36416" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:15.330011 31855 raft_consensus.cc:515] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.330080 31796 raft_consensus.cc:3060] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.330139 31855 leader_election.cc:290] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.330272 31646 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.330309 31646 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.330339 31721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.330403 31721 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.330529 31722 raft_consensus.cc:2468] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.330657 31611 leader_election.cc:304] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.330741 31883 raft_consensus.cc:2804] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.330778 31883 raft_consensus.cc:697] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:15.330813 31883 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.330864 31646 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 1.
I20251027 03:09:15.330998 31760 leader_election.cc:304] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:15.331060 31855 raft_consensus.cc:2804] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:15.330662 31796 raft_consensus.cc:2468] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 1.
I20251027 03:09:15.331113 31855 raft_consensus.cc:697] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:15.331274 31855 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.331143 31721 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 1.
I20251027 03:09:15.331682 31542 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.331766 31543 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:15.364921 31646 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.365135 31721 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.365242 31796 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.365340 31797 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.365350 31855 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.365409 31849 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.365502 31849 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.365603 31646 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.365706 31897 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.365968 31849 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.366183 31644 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.366262 31795 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.366423 31883 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.366669 31646 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.366937 31855 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.367091 31646 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.367144 31720 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.367244 31847 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.367362 31721 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.367388 31794 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.367249 31719 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.367560 31897 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.367594 31854 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.367676 31854 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.368044 31897 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.368132 31897 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.368609 31793 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:15.369412 31793 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369479 31720 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369513 31792 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369673 31641 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369787 31717 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369837 31639 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369916 31793 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369961 31791 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369913 31716 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.369998 31881 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.370019 31854 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.369791 31640 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.370179 31792 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:15.370909 31884 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371012 31884 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371073 31884 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371227 31847 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371302 31847 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371362 31847 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371417 31847 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371495 31847 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.371554 31847 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:15.375562 31920 mvcc.cc:204] Tried to move back new op lower bound from 7215245538779074560 to 7215245538291163136. Current Snapshot: MvccSnapshot[applied={T|T < 7215245538779074560}]
I20251027 03:09:15.378271 31919 mvcc.cc:204] Tried to move back new op lower bound from 7215245538782474240 to 7215245538637303808. Current Snapshot: MvccSnapshot[applied={T|T < 7215245538779643904}]
I20251027 03:09:15.381724 31923 mvcc.cc:204] Tried to move back new op lower bound from 7215245538787782656 to 7215245538635448320. Current Snapshot: MvccSnapshot[applied={T|T < 7215245538787782656 or (T in {7215245538787782656})}]
I20251027 03:09:15.467552 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:15.469338 32007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:15.469516 32006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:15.469663 32009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:15.471115 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:15.471242 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:15.471275 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:15.471292 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534555471292 us; error 0 us; skew 500 ppm
I20251027 03:09:15.471933 28253 webserver.cc:492] Webserver started at http://127.27.151.68:39997/ using document root <none> and password file <none>
I20251027 03:09:15.472005 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:15.472049 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:15.472110 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:15.472380 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "93e976cbba5e4213a43042ffe2da42d6"
format_stamp: "Formatted at 2025-10-27 03:09:15 on dist-test-slave-4cc8"
I20251027 03:09:15.473209 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.473693 32014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.473819 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.473863 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "93e976cbba5e4213a43042ffe2da42d6"
format_stamp: "Formatted at 2025-10-27 03:09:15 on dist-test-slave-4cc8"
I20251027 03:09:15.473924 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:15.489758 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:15.489928 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:15.490041 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:15.490232 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:15.490259 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.490289 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:15.490320 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.493563 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:44331
I20251027 03:09:15.493613 32077 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:44331 every 8 connection(s)
I20251027 03:09:15.493906 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:15.494822 32082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:15.494865 32083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:15.495059 32085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:15.495175 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:15.495270 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:15.495296 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:15.495313 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534555495313 us; error 0 us; skew 500 ppm
I20251027 03:09:15.495983 28253 webserver.cc:492] Webserver started at http://127.27.151.69:44707/ using document root <none> and password file <none>
I20251027 03:09:15.496068 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:15.496112 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:15.496186 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:15.496448 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-4-root/instance:
uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
format_stamp: "Formatted at 2025-10-27 03:09:15 on dist-test-slave-4cc8"
I20251027 03:09:15.497311 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.497676 32090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.497797 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.497840 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-4-root
uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
format_stamp: "Formatted at 2025-10-27 03:09:15 on dist-test-slave-4cc8"
I20251027 03:09:15.497879 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:15.499816 32078 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37795
I20251027 03:09:15.499871 32078 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:15.499972 32078 heartbeater.cc:507] Master 127.27.151.126:37795 requested a full tablet report, sending...
I20251027 03:09:15.500180 31542 ts_manager.cc:194] Registered new tserver with Master: 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331)
I20251027 03:09:15.500509 31542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48496
I20251027 03:09:15.502276 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:15.502455 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:15.502591 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:15.502758 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:15.502782 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.502813 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:15.502840 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.506220 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.69:42233
I20251027 03:09:15.506263 32153 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.69:42233 every 8 connection(s)
I20251027 03:09:15.506556 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:15.507476 32158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:15.507542 32160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:15.507776 32157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:15.508037 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:15.508136 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:15.508185 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:15.508210 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534555508211 us; error 0 us; skew 500 ppm
I20251027 03:09:15.508798 28253 webserver.cc:492] Webserver started at http://127.27.151.70:35643/ using document root <none> and password file <none>
I20251027 03:09:15.508875 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:15.508924 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:15.508970 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:15.509200 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-5-root/instance:
uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
format_stamp: "Formatted at 2025-10-27 03:09:15 on dist-test-slave-4cc8"
I20251027 03:09:15.509593 32154 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37795
I20251027 03:09:15.509671 32154 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:15.509764 32154 heartbeater.cc:507] Master 127.27.151.126:37795 requested a full tablet report, sending...
I20251027 03:09:15.509953 31542 ts_manager.cc:194] Registered new tserver with Master: bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233)
I20251027 03:09:15.510164 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:15.510293 31542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48512
I20251027 03:09:15.510620 32165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.510756 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.510797 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-5-root
uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
format_stamp: "Formatted at 2025-10-27 03:09:15 on dist-test-slave-4cc8"
I20251027 03:09:15.510861 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:15.537338 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:15.537530 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:15.537642 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:15.537801 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:15.537827 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.537856 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:15.537881 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:15.541015 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.70:42021
I20251027 03:09:15.541069 32228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.70:42021 every 8 connection(s)
I20251027 03:09:15.543931 32229 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37795
I20251027 03:09:15.543977 32229 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:15.544063 32229 heartbeater.cc:507] Master 127.27.151.126:37795 requested a full tablet report, sending...
I20251027 03:09:15.544279 31542 ts_manager.cc:194] Registered new tserver with Master: d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:15.544593 31542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48518
I20251027 03:09:15.859338 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:15.862320 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:15.862409 31640 raft_consensus.cc:606] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:15.862450 31795 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:15.862581 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:15.862641 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:15.862919 31717 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:15.862919 31640 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:15.863240 31995 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:15.863373 31998 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:15.863935 31998 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:15.864114 31640 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:15.865180 31542 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.865727 31717 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
W20251027 03:09:15.866611 31761 consensus_peers.cc:597] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e2464cf0e9243b7845c98d8cde8528b. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:15.867635 31640 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:15.868067 31717 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:15.868145 31795 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:15.868345 31975 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:15.868436 31978 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:15.868939 31883 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:15.869081 31795 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:15.869995 31540 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index -1 to 29, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.870249 31717 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:15.870520 31640 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:15.871198 31795 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:15.871286 31717 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:15.871359 31975 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:15.871462 31975 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20251027 03:09:15.871711 31612 consensus_peers.cc:597] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 -> Peer 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331): Couldn't send request to peer 93e976cbba5e4213a43042ffe2da42d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6045341ee60c472bbb01e952650a735a. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:15.872026 31978 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:15.872116 31717 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:15.872246 31795 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:15.873049 31542 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:15.873374 31795 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
W20251027 03:09:15.873827 31761 consensus_peers.cc:597] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 726dd98dfd6c47a5b95b723de71cb5c5. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:15.873966 31640 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:15.874006 31717 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:15.874164 31995 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:15.874250 31999 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:15.874763 31998 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:15.874876 31640 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:15.874930 31717 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:15.875836 31539 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index -1 to 28, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251027 03:09:15.876272 31592 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20251027 03:09:15.876390 31611 consensus_peers.cc:597] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 645f3c62551e42109ebfe785bde36416. This is attempt 1: this message will repeat every 5th retry.
W20251027 03:09:15.877281 31592 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 30ef3ff09aa247fab2eae47a3f1718fd, TS 50aa97f11c164e0b9e3520267cf6c85f -> TS 93e976cbba5e4213a43042ffe2da42d6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
W20251027 03:09:15.877318 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 30ef3ff09aa247fab2eae47a3f1718fd, TS 50aa97f11c164e0b9e3520267cf6c85f -> TS 93e976cbba5e4213a43042ffe2da42d6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
W20251027 03:09:15.878428 31592 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 11ce7fc7b4334ffe99df2944dca2af4a, TS 10153e76038242fdaec0d401cdbb0411 -> TS d421b303d20a48cfbbd34da50b8d0d6d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
W20251027 03:09:15.878464 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 11ce7fc7b4334ffe99df2944dca2af4a, TS 10153e76038242fdaec0d401cdbb0411 -> TS d421b303d20a48cfbbd34da50b8d0d6d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:15.947360 31977 raft_consensus.cc:993] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:15.947448 31977 raft_consensus.cc:1081] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:15.947683 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52978
I20251027 03:09:15.947758 31792 raft_consensus.cc:493] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:15.947786 31792 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:15.948378 31792 raft_consensus.cc:515] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.948503 31792 leader_election.cc:290] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:15.948666 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:15.948691 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:15.948760 31717 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:15.948812 31640 raft_consensus.cc:3055] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:15.948853 31640 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:15.948912 31640 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.948974 31640 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:15.949258 31717 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:15.949465 31762 leader_election.cc:304] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:15.949563 31999 raft_consensus.cc:2804] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:15.949630 31999 raft_consensus.cc:697] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:15.949642 31640 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:15.949676 31999 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:15.950198 31540 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 1 to 2, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 2 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:16.248047 32243 ts_tablet_manager.cc:933] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: Initiating tablet copy from peer 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:16.248270 32243 tablet_copy_client.cc:323] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:32889
I20251027 03:09:16.251343 31656 tablet_copy_service.cc:140] P 78e34a95a4a44fde91fa13a71fa3da26: Received BeginTabletCopySession request for tablet 6045341ee60c472bbb01e952650a735a from peer 93e976cbba5e4213a43042ffe2da42d6 ({username='slave'} at 127.0.0.1:43412)
I20251027 03:09:16.251417 31656 tablet_copy_service.cc:161] P 78e34a95a4a44fde91fa13a71fa3da26: Beginning new tablet copy session on tablet 6045341ee60c472bbb01e952650a735a from peer 93e976cbba5e4213a43042ffe2da42d6 at {username='slave'} at 127.0.0.1:43412: session id = 93e976cbba5e4213a43042ffe2da42d6-6045341ee60c472bbb01e952650a735a
I20251027 03:09:16.252112 31656 tablet_copy_source_session.cc:215] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:16.252480 32243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6045341ee60c472bbb01e952650a735a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:16.253691 32243 tablet_copy_client.cc:806] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:16.253829 32243 tablet_copy_client.cc:670] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:16.254304 32244 tablet_copy_client.cc:1025] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:16.311172 32246 ts_tablet_manager.cc:933] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Initiating tablet copy from peer 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:16.311390 32246 tablet_copy_client.cc:323] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:37045
I20251027 03:09:16.313472 32249 ts_tablet_manager.cc:933] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Initiating tablet copy from peer 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:16.313807 32249 tablet_copy_client.cc:323] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:32889
I20251027 03:09:16.314743 31807 tablet_copy_service.cc:140] P 10153e76038242fdaec0d401cdbb0411: Received BeginTabletCopySession request for tablet 0e2464cf0e9243b7845c98d8cde8528b from peer bcd3eff6ab6f42e9b6b475710ba14c80 ({username='slave'} at 127.0.0.1:52998)
I20251027 03:09:16.314796 31807 tablet_copy_service.cc:161] P 10153e76038242fdaec0d401cdbb0411: Beginning new tablet copy session on tablet 0e2464cf0e9243b7845c98d8cde8528b from peer bcd3eff6ab6f42e9b6b475710ba14c80 at {username='slave'} at 127.0.0.1:52998: session id = bcd3eff6ab6f42e9b6b475710ba14c80-0e2464cf0e9243b7845c98d8cde8528b
I20251027 03:09:16.315379 31807 tablet_copy_source_session.cc:215] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:16.315810 32246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2464cf0e9243b7845c98d8cde8528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:16.316677 31656 tablet_copy_service.cc:140] P 78e34a95a4a44fde91fa13a71fa3da26: Received BeginTabletCopySession request for tablet 645f3c62551e42109ebfe785bde36416 from peer d421b303d20a48cfbbd34da50b8d0d6d ({username='slave'} at 127.0.0.1:43414)
I20251027 03:09:16.316730 31656 tablet_copy_service.cc:161] P 78e34a95a4a44fde91fa13a71fa3da26: Beginning new tablet copy session on tablet 645f3c62551e42109ebfe785bde36416 from peer d421b303d20a48cfbbd34da50b8d0d6d at {username='slave'} at 127.0.0.1:43414: session id = d421b303d20a48cfbbd34da50b8d0d6d-645f3c62551e42109ebfe785bde36416
I20251027 03:09:16.317104 32246 tablet_copy_client.cc:806] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:16.317210 32246 tablet_copy_client.cc:670] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:16.317237 31656 tablet_copy_source_session.cc:215] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:16.317574 32249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645f3c62551e42109ebfe785bde36416. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:16.317795 32247 tablet_copy_client.cc:1025] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:16.318818 32249 tablet_copy_client.cc:806] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:16.318933 32249 tablet_copy_client.cc:670] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:16.319370 32250 tablet_copy_client.cc:1025] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:16.336648 31639 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:16.336813 31995 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:16.337862 31717 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:16.338048 32004 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:16.406368 32256 ts_tablet_manager.cc:933] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: Initiating tablet copy from peer 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:16.406534 32256 tablet_copy_client.cc:323] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:37045
I20251027 03:09:16.406692 31807 tablet_copy_service.cc:140] P 10153e76038242fdaec0d401cdbb0411: Received BeginTabletCopySession request for tablet 726dd98dfd6c47a5b95b723de71cb5c5 from peer bcd3eff6ab6f42e9b6b475710ba14c80 ({username='slave'} at 127.0.0.1:52998)
I20251027 03:09:16.406755 31807 tablet_copy_service.cc:161] P 10153e76038242fdaec0d401cdbb0411: Beginning new tablet copy session on tablet 726dd98dfd6c47a5b95b723de71cb5c5 from peer bcd3eff6ab6f42e9b6b475710ba14c80 at {username='slave'} at 127.0.0.1:52998: session id = bcd3eff6ab6f42e9b6b475710ba14c80-726dd98dfd6c47a5b95b723de71cb5c5
I20251027 03:09:16.407269 31807 tablet_copy_source_session.cc:215] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:16.407542 32256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726dd98dfd6c47a5b95b723de71cb5c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:16.408916 32256 tablet_copy_client.cc:806] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:16.409060 32256 tablet_copy_client.cc:670] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:16.409574 32257 tablet_copy_client.cc:1025] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:16.501019 32078 heartbeater.cc:499] Master 127.27.151.126:37795 was elected leader, sending a full tablet report...
I20251027 03:09:16.510752 32154 heartbeater.cc:499] Master 127.27.151.126:37795 was elected leader, sending a full tablet report...
I20251027 03:09:16.545002 32229 heartbeater.cc:499] Master 127.27.151.126:37795 was elected leader, sending a full tablet report...
I20251027 03:09:17.862835 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:17.866205 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:17.866290 31795 raft_consensus.cc:606] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:17.866430 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:17.866482 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:18.088570 31999 raft_consensus.cc:993] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:18.088690 31999 raft_consensus.cc:1081] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:18.088922 31640 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43380
I20251027 03:09:18.088997 31640 raft_consensus.cc:493] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:18.089035 31640 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:18.089825 31640 raft_consensus.cc:515] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.090001 31640 leader_election.cc:290] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 2 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:18.090108 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:18.090204 31717 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:18.090730 31717 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 2.
I20251027 03:09:18.090873 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:18.090896 31611 leader_election.cc:304] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:18.090956 31795 raft_consensus.cc:3055] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:18.090984 31795 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:18.091006 32258 raft_consensus.cc:2804] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:18.091034 31795 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.091101 32258 raft_consensus.cc:697] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:18.091126 31795 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:18.091154 32258 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.091652 31795 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 2.
I20251027 03:09:18.091715 31542 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 1 to 2, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 2 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:18.093677 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:18.093734 31640 raft_consensus.cc:606] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Received request to transfer leadership
I20251027 03:09:18.094777 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:18.094831 31640 raft_consensus.cc:606] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Received request to transfer leadership
W20251027 03:09:18.094951 31527 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 0e2464cf0e9243b7845c98d8cde8528b already in progress [suppressed 2 similar messages]
I20251027 03:09:18.095229 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:18.095271 31640 raft_consensus.cc:606] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Received request to transfer leadership
W20251027 03:09:18.095394 31592 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 0e2464cf0e9243b7845c98d8cde8528b already in progress
W20251027 03:09:18.095440 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 0e2464cf0e9243b7845c98d8cde8528b already in progress
I20251027 03:09:18.254508 32243 tablet_copy_client.cc:538] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:18.255400 32243 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap starting.
I20251027 03:09:18.260277 32243 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:18.260488 32243 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap complete.
I20251027 03:09:18.260618 32243 ts_tablet_manager.cc:1403] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251027 03:09:18.260788 32243 raft_consensus.cc:359] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:18.260880 32243 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Initialized, Role: LEARNER
I20251027 03:09:18.260917 32243 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:18.261085 32243 ts_tablet_manager.cc:1434] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:09:18.261422 31656 tablet_copy_service.cc:342] P 78e34a95a4a44fde91fa13a71fa3da26: Request end of tablet copy session 93e976cbba5e4213a43042ffe2da42d6-6045341ee60c472bbb01e952650a735a received from {username='slave'} at 127.0.0.1:43412
I20251027 03:09:18.261478 31656 tablet_copy_service.cc:434] P 78e34a95a4a44fde91fa13a71fa3da26: ending tablet copy session 93e976cbba5e4213a43042ffe2da42d6-6045341ee60c472bbb01e952650a735a on tablet 6045341ee60c472bbb01e952650a735a with peer 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:18.300079 32052 raft_consensus.cc:1217] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20251027 03:09:18.318014 32246 tablet_copy_client.cc:538] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:18.318768 32246 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap starting.
I20251027 03:09:18.319568 32249 tablet_copy_client.cc:538] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:18.320299 32249 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap starting.
I20251027 03:09:18.323312 32246 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=119 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:18.323551 32246 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap complete.
I20251027 03:09:18.323695 32246 ts_tablet_manager.cc:1403] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.004s
I20251027 03:09:18.323889 32246 raft_consensus.cc:359] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.323978 32246 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Initialized, Role: LEARNER
I20251027 03:09:18.324016 32246 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.324254 32246 ts_tablet_manager.cc:1434] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:18.324488 31807 tablet_copy_service.cc:342] P 10153e76038242fdaec0d401cdbb0411: Request end of tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-0e2464cf0e9243b7845c98d8cde8528b received from {username='slave'} at 127.0.0.1:52998
I20251027 03:09:18.324542 31807 tablet_copy_service.cc:434] P 10153e76038242fdaec0d401cdbb0411: ending tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-0e2464cf0e9243b7845c98d8cde8528b on tablet 0e2464cf0e9243b7845c98d8cde8528b with peer bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:18.325124 32249 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:18.325309 32249 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap complete.
I20251027 03:09:18.325403 32249 ts_tablet_manager.cc:1403] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251027 03:09:18.325539 32249 raft_consensus.cc:359] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:18.325614 32249 raft_consensus.cc:740] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Initialized, Role: LEARNER
I20251027 03:09:18.325642 32249 consensus_queue.cc:260] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:18.325780 32249 ts_tablet_manager.cc:1434] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Time spent starting tablet: real 0.000s user 0.001s sys 0.001s
I20251027 03:09:18.325954 31656 tablet_copy_service.cc:342] P 78e34a95a4a44fde91fa13a71fa3da26: Request end of tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-645f3c62551e42109ebfe785bde36416 received from {username='slave'} at 127.0.0.1:43414
I20251027 03:09:18.326020 31656 tablet_copy_service.cc:434] P 78e34a95a4a44fde91fa13a71fa3da26: ending tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-645f3c62551e42109ebfe785bde36416 on tablet 645f3c62551e42109ebfe785bde36416 with peer d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:18.409781 32256 tablet_copy_client.cc:538] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:18.410836 32256 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap starting.
I20251027 03:09:18.414747 32256 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:18.414943 32256 tablet_bootstrap.cc:492] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap complete.
I20251027 03:09:18.415047 32256 ts_tablet_manager.cc:1403] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251027 03:09:18.415396 32256 raft_consensus.cc:359] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.415511 32256 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Initialized, Role: LEARNER
I20251027 03:09:18.415551 32256 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.415640 32256 ts_tablet_manager.cc:1434] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20251027 03:09:18.415805 31807 tablet_copy_service.cc:342] P 10153e76038242fdaec0d401cdbb0411: Request end of tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-726dd98dfd6c47a5b95b723de71cb5c5 received from {username='slave'} at 127.0.0.1:52998
I20251027 03:09:18.415849 31807 tablet_copy_service.cc:434] P 10153e76038242fdaec0d401cdbb0411: ending tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-726dd98dfd6c47a5b95b723de71cb5c5 on tablet 726dd98dfd6c47a5b95b723de71cb5c5 with peer bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:18.581674 31792 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251027 03:09:18.581926 32258 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:18.582804 32263 raft_consensus.cc:993] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:18.582883 32263 raft_consensus.cc:1081] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:18.583258 31717 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251027 03:09:18.583510 32281 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:18.583947 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52978
I20251027 03:09:18.584003 31792 raft_consensus.cc:493] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:18.584028 31792 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:18.584800 31792 raft_consensus.cc:515] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.584949 31792 leader_election.cc:290] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:18.585037 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:18.585095 31640 raft_consensus.cc:3055] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:18.585110 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:18.585119 31640 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:18.585170 31717 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:18.585242 31640 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.585315 31640 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:18.585718 31717 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:18.585904 31762 leader_election.cc:304] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:18.585976 31999 raft_consensus.cc:2804] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:18.586009 31999 raft_consensus.cc:697] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:18.586030 31640 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:18.586054 31999 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:18.586561 31540 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 2 to 3, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:18.586716 32128 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Advancing to term 2
I20251027 03:09:18.587492 32128 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 2 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251027 03:09:18.757656 32128 raft_consensus.cc:1217] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20251027 03:09:18.816526 32203 raft_consensus.cc:1217] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20251027 03:09:18.843390 32281 raft_consensus.cc:1064] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:18.843596 32281 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:18.844058 31717 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:18.844058 32052 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:18.844249 32263 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:18.844578 31792 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:18.846402 32280 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:18.846625 32281 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:18.847380 32281 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.847532 31717 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.848215 31792 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.848512 32052 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.849493 31540 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index 29 to 30, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.850339 31639 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:18.850787 32052 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:18.850907 31792 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:18.850986 32263 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:18.851099 32280 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:18.851454 32263 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.851594 31792 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.851594 32052 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:18.852283 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6045341ee60c472bbb01e952650a735a with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:18.852406 31542 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 30 to 31, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:18.853103 31704 tablet_service.cc:1552] Processing DeleteTablet for tablet 6045341ee60c472bbb01e952650a735a with delete_type TABLET_DATA_TOMBSTONED (TS 50aa97f11c164e0b9e3520267cf6c85f not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:18.853401 32293 tablet_replica.cc:333] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:18.853509 32293 raft_consensus.cc:2243] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:18.853637 32293 raft_consensus.cc:2272] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:18.853873 32293 ts_tablet_manager.cc:1916] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:18.855130 32293 ts_tablet_manager.cc:1929] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20251027 03:09:18.855186 32293 log.cc:1199] T 6045341ee60c472bbb01e952650a735a P 50aa97f11c164e0b9e3520267cf6c85f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/6045341ee60c472bbb01e952650a735a
I20251027 03:09:18.855545 31529 catalog_manager.cc:4985] TS 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091): tablet 6045341ee60c472bbb01e952650a735a (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:18.986886 32128 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 2 LEARNER]: Advancing to term 3
I20251027 03:09:18.987769 32128 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:18.988026 32287 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:18.988911 31999 raft_consensus.cc:1064] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
W20251027 03:09:18.988967 31999 raft_consensus.cc:1067] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: Unable to promote non-voter bcd3eff6ab6f42e9b6b475710ba14c80: Illegal state: Leader has not yet committed an operation in its own term
I20251027 03:09:19.146996 31639 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:19.147231 31999 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:19.148211 31717 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:19.148504 32262 raft_consensus.cc:1064] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
I20251027 03:09:19.148689 32262 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:19.149272 32128 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:19.149444 31717 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:19.149461 31640 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:19.149844 32298 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:19.149988 31999 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:19.150122 32287 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:19.150740 32287 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.151364 31640 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.152012 32128 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.152554 31539 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 29 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.153684 31795 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:19.153690 31717 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.154570 31719 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251027 03:09:19.154805 32300 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:19.154942 32128 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251027 03:09:19.155114 32300 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:19.155403 32287 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.155566 31719 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.156353 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0e2464cf0e9243b7845c98d8cde8528b with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:19.156755 31538 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:19.157231 32128 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.158327 31635 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e2464cf0e9243b7845c98d8cde8528b with delete_type TABLET_DATA_TOMBSTONED (TS 78e34a95a4a44fde91fa13a71fa3da26 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:19.158574 32302 tablet_replica.cc:333] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:19.158643 32302 raft_consensus.cc:2243] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:19.158754 32302 raft_consensus.cc:2272] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:19.159022 32302 ts_tablet_manager.cc:1916] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:19.160425 32302 ts_tablet_manager.cc:1929] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20251027 03:09:19.160487 32302 log.cc:1199] T 0e2464cf0e9243b7845c98d8cde8528b P 78e34a95a4a44fde91fa13a71fa3da26: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/0e2464cf0e9243b7845c98d8cde8528b
I20251027 03:09:19.160883 31527 catalog_manager.cc:4985] TS 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889): tablet 0e2464cf0e9243b7845c98d8cde8528b (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:19.216087 32258 raft_consensus.cc:1064] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d to VOTER
I20251027 03:09:19.216329 32258 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:19.216823 32203 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:19.216846 31719 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:19.217029 31795 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:19.217048 32281 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:19.217176 32263 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:19.217370 32263 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:19.218055 32263 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.218185 32203 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.218385 31792 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.219235 31539 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: config changed from index 29 to 30, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.219573 31719 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.220201 31640 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:19.220742 32203 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:19.220767 31719 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:19.220958 32281 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:19.221048 32280 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:19.221623 32280 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.222390 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 645f3c62551e42109ebfe785bde36416 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:19.222525 32203 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.222528 31538 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 30 to 31, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:19.223299 31719 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:19.223352 31779 tablet_service.cc:1552] Processing DeleteTablet for tablet 645f3c62551e42109ebfe785bde36416 with delete_type TABLET_DATA_TOMBSTONED (TS 10153e76038242fdaec0d401cdbb0411 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52962
I20251027 03:09:19.223691 32307 tablet_replica.cc:333] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:19.223750 32307 raft_consensus.cc:2243] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:19.223820 32307 raft_consensus.cc:2272] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:19.224098 32307 ts_tablet_manager.cc:1916] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:19.225222 32307 ts_tablet_manager.cc:1929] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20251027 03:09:19.225286 32307 log.cc:1199] T 645f3c62551e42109ebfe785bde36416 P 10153e76038242fdaec0d401cdbb0411: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/645f3c62551e42109ebfe785bde36416
I20251027 03:09:19.225656 31527 catalog_manager.cc:4985] TS 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045): tablet 645f3c62551e42109ebfe785bde36416 (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:19.373759 32298 raft_consensus.cc:1064] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
I20251027 03:09:19.374109 32298 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:19.374640 31719 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:19.374640 31639 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:19.374884 32128 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:19.374859 32262 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:19.375015 32300 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:19.375131 32262 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:19.375859 32287 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.376037 32128 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.376165 31639 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.377082 31719 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.377162 31542 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 28 to 29, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.378212 31792 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:19.378504 31719 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:19.378558 32128 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:19.378690 32262 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:19.378769 32300 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:19.379283 32299 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.379364 31719 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.380271 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 726dd98dfd6c47a5b95b723de71cb5c5 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:19.380280 31539 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 29 to 30, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New cstate: current_term: 1 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:19.380357 32128 raft_consensus.cc:2955] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:19.381026 31635 tablet_service.cc:1552] Processing DeleteTablet for tablet 726dd98dfd6c47a5b95b723de71cb5c5 with delete_type TABLET_DATA_TOMBSTONED (TS 78e34a95a4a44fde91fa13a71fa3da26 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:19.381130 32302 tablet_replica.cc:333] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:19.381209 32302 raft_consensus.cc:2243] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:19.381279 32302 raft_consensus.cc:2272] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:19.381546 32302 ts_tablet_manager.cc:1916] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:19.382498 32302 ts_tablet_manager.cc:1929] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20251027 03:09:19.382560 32302 log.cc:1199] T 726dd98dfd6c47a5b95b723de71cb5c5 P 78e34a95a4a44fde91fa13a71fa3da26: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/726dd98dfd6c47a5b95b723de71cb5c5
I20251027 03:09:19.382875 31527 catalog_manager.cc:4985] TS 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889): tablet 726dd98dfd6c47a5b95b723de71cb5c5 (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:19.866613 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:19.866984 31792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:19.867064 31792 raft_consensus.cc:606] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Received request to transfer leadership to TS bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:19.869714 31719 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:19.869782 31719 raft_consensus.cc:606] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:19.870110 31792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:19.870178 31792 raft_consensus.cc:606] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:19.870453 31719 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:19.870515 31719 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:19.870826 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "645f3c62551e42109ebfe785bde36416"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:19.870878 31639 raft_consensus.cc:606] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Received request to transfer leadership to TS d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:19.871040 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20251027 03:09:19.871114 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:20.045773 32300 raft_consensus.cc:993] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:20.045857 32300 raft_consensus.cc:1081] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:20.046041 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43380
I20251027 03:09:20.046128 31639 raft_consensus.cc:493] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:20.046159 31639 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:20.046778 31639 raft_consensus.cc:515] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.046952 31639 leader_election.cc:290] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:20.047086 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:20.047147 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30ef3ff09aa247fab2eae47a3f1718fd" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:20.047163 31719 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:20.047199 31792 raft_consensus.cc:3055] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:20.047223 31792 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:20.047284 31792 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.047364 31792 raft_consensus.cc:3060] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:20.047914 31719 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:20.048025 31792 raft_consensus.cc:2468] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:20.048127 31611 leader_election.cc:304] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:20.048228 32281 raft_consensus.cc:2804] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:20.048322 32281 raft_consensus.cc:697] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:20.048384 32281 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.048863 31539 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 2 to 3, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:20.113229 32315 raft_consensus.cc:993] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower d421b303d20a48cfbbd34da50b8d0d6d to start an election
I20251027 03:09:20.113309 32315 raft_consensus.cc:1081] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Signalling peer d421b303d20a48cfbbd34da50b8d0d6d to start an election
I20251027 03:09:20.113502 32203 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "645f3c62551e42109ebfe785bde36416"
dest_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
from {username='slave'} at 127.0.0.1:50822
I20251027 03:09:20.113577 32203 raft_consensus.cc:493] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:20.113596 32203 raft_consensus.cc:3060] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.114161 32203 raft_consensus.cc:515] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:20.114305 32203 leader_election.cc:290] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:20.114462 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645f3c62551e42109ebfe785bde36416" candidate_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:20.114530 31639 raft_consensus.cc:3055] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:20.114552 31639 raft_consensus.cc:740] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:20.114600 31639 consensus_queue.cc:260] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:20.114687 31639 raft_consensus.cc:3060] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.115198 31639 raft_consensus.cc:2468] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d421b303d20a48cfbbd34da50b8d0d6d in term 2.
I20251027 03:09:20.115357 32166 leader_election.cc:304] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78e34a95a4a44fde91fa13a71fa3da26, d421b303d20a48cfbbd34da50b8d0d6d; no voters:
I20251027 03:09:20.115556 32321 raft_consensus.cc:2804] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:20.115672 32321 raft_consensus.cc:697] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEADER]: Becoming Leader. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Running, Role: LEADER
I20251027 03:09:20.115729 32321 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:20.116204 31539 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: term changed from 1 to 2, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70). New cstate: current_term: 2 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:20.117142 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "645f3c62551e42109ebfe785bde36416" candidate_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:20.117214 31719 raft_consensus.cc:3060] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.117449 32286 raft_consensus.cc:993] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:20.117491 32286 raft_consensus.cc:1081] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:20.117838 31719 raft_consensus.cc:2468] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d421b303d20a48cfbbd34da50b8d0d6d in term 2.
I20251027 03:09:20.118495 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:20.118564 31792 raft_consensus.cc:493] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:20.118597 31792 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.119204 31792 raft_consensus.cc:515] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:20.119335 31792 leader_election.cc:290] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:20.119436 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:20.119511 31639 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.120183 31639 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:20.120306 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:20.120368 31719 raft_consensus.cc:3055] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:20.120371 31760 leader_election.cc:304] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:20.120407 31719 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:20.120488 32300 raft_consensus.cc:2804] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:20.120528 32300 raft_consensus.cc:697] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:20.120523 31719 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:20.120620 32300 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
I20251027 03:09:20.120643 31719 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.121294 31719 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:20.121258 31539 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 1 to 2, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 2 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:20.211421 32300 raft_consensus.cc:993] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: : Instructing follower bcd3eff6ab6f42e9b6b475710ba14c80 to start an election
I20251027 03:09:20.211505 32300 raft_consensus.cc:1081] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Signalling peer bcd3eff6ab6f42e9b6b475710ba14c80 to start an election
I20251027 03:09:20.211700 32128 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b"
dest_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
from {username='slave'} at 127.0.0.1:49686
I20251027 03:09:20.211767 32128 raft_consensus.cc:493] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:20.211786 32128 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:20.212374 32128 raft_consensus.cc:515] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:20.212517 32128 leader_election.cc:290] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [CANDIDATE]: Term 4 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:20.212684 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:20.212776 31792 raft_consensus.cc:3055] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:20.212805 31792 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:20.212850 31792 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:20.212934 31792 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:20.213560 31792 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd3eff6ab6f42e9b6b475710ba14c80 in term 4.
I20251027 03:09:20.213753 32091 leader_election.cc:304] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, bcd3eff6ab6f42e9b6b475710ba14c80; no voters:
I20251027 03:09:20.213901 32326 raft_consensus.cc:2804] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:20.214016 32326 raft_consensus.cc:697] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEADER]: Becoming Leader. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Running, Role: LEADER
I20251027 03:09:20.214082 32326 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:20.214622 31539 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 reported cstate change: term changed from 3 to 4, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69). New cstate: current_term: 4 leader_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:20.215363 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2464cf0e9243b7845c98d8cde8528b" candidate_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:20.215417 31719 raft_consensus.cc:3060] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:20.215874 31719 raft_consensus.cc:2468] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcd3eff6ab6f42e9b6b475710ba14c80 in term 4.
I20251027 03:09:20.253198 32323 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:20.253273 32323 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:20.253415 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:20.253490 31792 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:20.253510 31792 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.254061 31792 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.254181 31792 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:20.254300 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:20.254313 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:20.254370 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.254410 31719 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:20.254442 31719 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:20.254500 31719 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.254585 31719 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:20.255020 31639 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:20.255095 31719 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:20.255230 31760 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:20.255317 32300 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:20.255380 32300 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:20.255436 32300 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.255920 31539 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 1 to 2, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 2 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:20.383751 31719 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:20.384256 31639 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251027 03:09:20.384311 31792 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251027 03:09:20.384465 32286 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251027 03:09:20.384622 32332 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251027 03:09:20.385136 32286 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:20.385299 31639 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:20.385360 31792 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:20.386166 31539 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index -1 to 27, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:20.386533 31639 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
W20251027 03:09:20.387079 31611 consensus_peers.cc:597] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 914edf6a5ed5402fb1d07b6afba645b6. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:20.387107 31792 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:20.387246 32315 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:20.387388 31719 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:20.387593 32315 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:20.388235 32315 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
W20251027 03:09:20.388368 31688 consensus_peers.cc:597] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f -> Peer 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331): Couldn't send request to peer 93e976cbba5e4213a43042ffe2da42d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c176d9f872b84ac986522990949395eb. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:20.388404 31719 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:20.389353 31538 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index -1 to 28, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:20.389513 31639 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:20.389590 31792 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:20.390000 31795 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
W20251027 03:09:20.390028 31611 consensus_peers.cc:597] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6045341ee60c472bbb01e952650a735a. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:20.390239 32342 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:20.390525 32052 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251027 03:09:20.390825 32342 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:20.391016 32315 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:20.391521 32052 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:20.391839 31792 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
W20251027 03:09:20.392096 31592 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20251027 03:09:20.392604 31542 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 31 to 32, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251027 03:09:20.393546 31592 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 11ce7fc7b4334ffe99df2944dca2af4a, TS 78e34a95a4a44fde91fa13a71fa3da26 -> TS 93e976cbba5e4213a43042ffe2da42d6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
W20251027 03:09:20.393589 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 11ce7fc7b4334ffe99df2944dca2af4a, TS 78e34a95a4a44fde91fa13a71fa3da26 -> TS 93e976cbba5e4213a43042ffe2da42d6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } }
W20251027 03:09:20.394430 31592 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 42bb6d5c4d0e42af9c337417582580b6, TS 50aa97f11c164e0b9e3520267cf6c85f -> TS bcd3eff6ab6f42e9b6b475710ba14c80 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
W20251027 03:09:20.394476 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 42bb6d5c4d0e42af9c337417582580b6, TS 50aa97f11c164e0b9e3520267cf6c85f -> TS bcd3eff6ab6f42e9b6b475710ba14c80 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
W20251027 03:09:20.395339 31592 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet bb4aebbafbe6415e90eaf6f6feb6b760, TS 78e34a95a4a44fde91fa13a71fa3da26 -> TS d421b303d20a48cfbbd34da50b8d0d6d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
W20251027 03:09:20.395380 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet bb4aebbafbe6415e90eaf6f6feb6b760, TS 78e34a95a4a44fde91fa13a71fa3da26 -> TS d421b303d20a48cfbbd34da50b8d0d6d move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:20.476006 31792 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:20.476274 32342 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:20.477427 31719 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:20.477589 32342 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:20.625017 31639 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251027 03:09:20.625273 32351 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:20.626339 31717 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251027 03:09:20.626593 32356 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:20.671537 31639 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:20.671792 32300 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:20.672796 31717 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:20.672986 32314 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:20.680341 31717 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:20.680486 32314 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:20.681308 31639 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:20.681457 32300 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:20.718259 31792 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20251027 03:09:20.718468 32362 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:20.719589 31719 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20251027 03:09:20.719802 32362 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:20.860080 32368 ts_tablet_manager.cc:933] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: Initiating tablet copy from peer 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:20.860304 32368 tablet_copy_client.cc:323] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:41091
I20251027 03:09:20.863338 31732 tablet_copy_service.cc:140] P 50aa97f11c164e0b9e3520267cf6c85f: Received BeginTabletCopySession request for tablet c176d9f872b84ac986522990949395eb from peer 93e976cbba5e4213a43042ffe2da42d6 ({username='slave'} at 127.0.0.1:34902)
I20251027 03:09:20.863392 31732 tablet_copy_service.cc:161] P 50aa97f11c164e0b9e3520267cf6c85f: Beginning new tablet copy session on tablet c176d9f872b84ac986522990949395eb from peer 93e976cbba5e4213a43042ffe2da42d6 at {username='slave'} at 127.0.0.1:34902: session id = 93e976cbba5e4213a43042ffe2da42d6-c176d9f872b84ac986522990949395eb
I20251027 03:09:20.863863 31732 tablet_copy_source_session.cc:215] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:20.864210 32368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176d9f872b84ac986522990949395eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:20.865774 32368 tablet_copy_client.cc:806] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:20.865942 32368 tablet_copy_client.cc:670] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:20.866433 32369 tablet_copy_client.cc:1025] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:20.901022 32371 ts_tablet_manager.cc:933] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: Initiating tablet copy from peer 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:20.901201 32371 tablet_copy_client.cc:323] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:32889
I20251027 03:09:20.901450 31656 tablet_copy_service.cc:140] P 78e34a95a4a44fde91fa13a71fa3da26: Received BeginTabletCopySession request for tablet 6045341ee60c472bbb01e952650a735a from peer d421b303d20a48cfbbd34da50b8d0d6d ({username='slave'} at 127.0.0.1:43414)
I20251027 03:09:20.901516 31656 tablet_copy_service.cc:161] P 78e34a95a4a44fde91fa13a71fa3da26: Beginning new tablet copy session on tablet 6045341ee60c472bbb01e952650a735a from peer d421b303d20a48cfbbd34da50b8d0d6d at {username='slave'} at 127.0.0.1:43414: session id = d421b303d20a48cfbbd34da50b8d0d6d-6045341ee60c472bbb01e952650a735a
I20251027 03:09:20.901976 31656 tablet_copy_source_session.cc:215] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:20.902285 32371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6045341ee60c472bbb01e952650a735a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:20.903218 32371 tablet_copy_client.cc:806] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:20.903373 32371 tablet_copy_client.cc:670] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:20.903885 32372 tablet_copy_client.cc:1025] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:21.011638 32373 ts_tablet_manager.cc:933] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: Initiating tablet copy from peer 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:21.011932 32373 tablet_copy_client.cc:323] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:32889
I20251027 03:09:21.012107 31656 tablet_copy_service.cc:140] P 78e34a95a4a44fde91fa13a71fa3da26: Received BeginTabletCopySession request for tablet 914edf6a5ed5402fb1d07b6afba645b6 from peer d421b303d20a48cfbbd34da50b8d0d6d ({username='slave'} at 127.0.0.1:43414)
I20251027 03:09:21.012195 31656 tablet_copy_service.cc:161] P 78e34a95a4a44fde91fa13a71fa3da26: Beginning new tablet copy session on tablet 914edf6a5ed5402fb1d07b6afba645b6 from peer d421b303d20a48cfbbd34da50b8d0d6d at {username='slave'} at 127.0.0.1:43414: session id = d421b303d20a48cfbbd34da50b8d0d6d-914edf6a5ed5402fb1d07b6afba645b6
I20251027 03:09:21.012634 31656 tablet_copy_source_session.cc:215] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:21.012979 32373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 914edf6a5ed5402fb1d07b6afba645b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:21.014052 32373 tablet_copy_client.cc:806] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:21.014199 32373 tablet_copy_client.cc:670] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:21.014719 32374 tablet_copy_client.cc:1025] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:21.871261 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:21.871670 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:21.871778 31717 raft_consensus.cc:606] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:21.872103 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:21.872197 31640 raft_consensus.cc:606] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:21.872514 31792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:21.872586 31792 raft_consensus.cc:606] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:21.872745 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251027 03:09:21.872814 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:22.009825 32378 raft_consensus.cc:993] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:22.009936 32378 raft_consensus.cc:1081] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:22.010123 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:22.010190 31717 raft_consensus.cc:493] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:22.010213 31717 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.011045 31717 raft_consensus.cc:515] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.011184 31717 leader_election.cc:290] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:22.011339 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:22.011363 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:22.011437 31792 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.011482 31640 raft_consensus.cc:3055] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:22.011513 31640 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:22.011560 31640 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.011652 31640 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.012269 31792 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 2.
I20251027 03:09:22.012413 31685 leader_election.cc:304] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:22.012435 31640 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 2.
I20251027 03:09:22.012483 32382 raft_consensus.cc:2804] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:22.012527 32382 raft_consensus.cc:697] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:22.012580 32382 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.013197 31539 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 1 to 2, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 2 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:22.014524 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:22.014578 31717 raft_consensus.cc:606] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Received request to transfer leadership
I20251027 03:09:22.015431 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:22.015479 31717 raft_consensus.cc:606] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Received request to transfer leadership
I20251027 03:09:22.015798 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:22.015841 31717 raft_consensus.cc:606] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Received request to transfer leadership
W20251027 03:09:22.015965 31592 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 914edf6a5ed5402fb1d07b6afba645b6 already in progress
W20251027 03:09:22.016006 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 914edf6a5ed5402fb1d07b6afba645b6 already in progress
I20251027 03:09:22.142247 32314 raft_consensus.cc:993] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:22.142338 32314 raft_consensus.cc:1081] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:22.142485 31719 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57726
I20251027 03:09:22.142575 31719 raft_consensus.cc:493] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:22.142611 31719 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.143381 31719 raft_consensus.cc:515] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:22.143498 31719 leader_election.cc:290] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:22.143617 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:22.143718 31792 raft_consensus.cc:3055] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:22.143754 31792 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:22.143801 31792 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:22.143874 31792 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.144542 31792 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 2.
I20251027 03:09:22.144680 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:22.144677 31685 leader_election.cc:304] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:22.144765 31640 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.144807 32382 raft_consensus.cc:2804] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:22.144852 32382 raft_consensus.cc:697] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:22.144922 32382 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:22.145450 31640 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 2.
I20251027 03:09:22.145581 31539 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 1 to 2, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 2 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:22.263091 32381 raft_consensus.cc:993] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:22.263185 32381 raft_consensus.cc:1081] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:22.263370 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:22.263437 31792 raft_consensus.cc:493] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:22.263458 31792 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.264081 31792 raft_consensus.cc:515] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:22.264230 31792 leader_election.cc:290] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:22.264400 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:22.264420 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:22.264505 31719 raft_consensus.cc:3055] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:22.264541 31719 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:22.264600 31719 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:22.264680 31719 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.265358 31719 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:22.264468 31640 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:22.265489 31762 leader_election.cc:304] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:22.265604 32314 raft_consensus.cc:2804] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:22.265671 32314 raft_consensus.cc:697] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:22.265717 32314 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:22.266199 31640 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 2.
I20251027 03:09:22.266254 31539 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 1 to 2, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 2 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:22.424453 31792 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:22.424676 32381 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:22.425477 32381 raft_consensus.cc:993] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:22.425530 32381 raft_consensus.cc:1081] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:22.425920 31640 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:22.426101 32332 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:22.426105 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:22.426208 31792 raft_consensus.cc:493] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:22.426235 31792 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:22.427080 31792 raft_consensus.cc:515] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.427213 31792 leader_election.cc:290] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:22.427290 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:22.427290 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:22.427376 31640 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:22.427384 31719 raft_consensus.cc:3055] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:22.427421 31719 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:22.427470 31719 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.427539 31719 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:22.428036 31640 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:22.428172 31719 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:22.428305 31762 leader_election.cc:304] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:22.428380 32314 raft_consensus.cc:2804] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:22.428445 32314 raft_consensus.cc:697] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:22.428493 32314 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.428978 31539 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 2 to 3, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:22.593238 31792 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:22.593415 32382 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:22.594456 31639 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:22.594626 32381 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:22.818714 31640 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251027 03:09:22.818939 32314 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:22.820166 31719 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251027 03:09:22.820375 32314 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:22.866617 32368 tablet_copy_client.cc:538] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:22.867563 32368 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap starting.
I20251027 03:09:22.871122 32368 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:22.871320 32368 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap complete.
I20251027 03:09:22.871428 32368 ts_tablet_manager.cc:1403] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251027 03:09:22.871589 32368 raft_consensus.cc:359] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:22.871675 32368 raft_consensus.cc:740] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Initialized, Role: LEARNER
I20251027 03:09:22.871707 32368 consensus_queue.cc:260] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:22.871888 32368 ts_tablet_manager.cc:1434] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: Time spent starting tablet: real 0.000s user 0.002s sys 0.001s
I20251027 03:09:22.872187 31732 tablet_copy_service.cc:342] P 50aa97f11c164e0b9e3520267cf6c85f: Request end of tablet copy session 93e976cbba5e4213a43042ffe2da42d6-c176d9f872b84ac986522990949395eb received from {username='slave'} at 127.0.0.1:34902
I20251027 03:09:22.872236 31732 tablet_copy_service.cc:434] P 50aa97f11c164e0b9e3520267cf6c85f: ending tablet copy session 93e976cbba5e4213a43042ffe2da42d6-c176d9f872b84ac986522990949395eb on tablet c176d9f872b84ac986522990949395eb with peer 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:22.904098 32371 tablet_copy_client.cc:538] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:22.904790 32371 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap starting.
I20251027 03:09:22.908789 32371 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=122 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:22.908980 32371 tablet_bootstrap.cc:492] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap complete.
I20251027 03:09:22.909078 32371 ts_tablet_manager.cc:1403] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20251027 03:09:22.909230 32371 raft_consensus.cc:359] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.909313 32371 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Initialized, Role: LEARNER
I20251027 03:09:22.909346 32371 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:22.909446 32371 ts_tablet_manager.cc:1434] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:22.909614 31656 tablet_copy_service.cc:342] P 78e34a95a4a44fde91fa13a71fa3da26: Request end of tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-6045341ee60c472bbb01e952650a735a received from {username='slave'} at 127.0.0.1:43414
I20251027 03:09:22.909662 31656 tablet_copy_service.cc:434] P 78e34a95a4a44fde91fa13a71fa3da26: ending tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-6045341ee60c472bbb01e952650a735a on tablet 6045341ee60c472bbb01e952650a735a with peer d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:22.994429 31719 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:22.994616 32314 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:22.995945 31639 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:22.996124 32389 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20251027 03:09:22.999297 31762 consensus_peers.cc:597] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:23.014926 32373 tablet_copy_client.cc:538] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:23.015779 32373 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap starting.
I20251027 03:09:23.019455 32373 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:23.019646 32373 tablet_bootstrap.cc:492] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap complete.
I20251027 03:09:23.019762 32373 ts_tablet_manager.cc:1403] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251027 03:09:23.019928 32373 raft_consensus.cc:359] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:23.020056 32373 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Initialized, Role: LEARNER
I20251027 03:09:23.020109 32373 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:23.020263 32373 ts_tablet_manager.cc:1434] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20251027 03:09:23.020416 31656 tablet_copy_service.cc:342] P 78e34a95a4a44fde91fa13a71fa3da26: Request end of tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-914edf6a5ed5402fb1d07b6afba645b6 received from {username='slave'} at 127.0.0.1:43414
I20251027 03:09:23.020468 31656 tablet_copy_service.cc:434] P 78e34a95a4a44fde91fa13a71fa3da26: ending tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-914edf6a5ed5402fb1d07b6afba645b6 on tablet 914edf6a5ed5402fb1d07b6afba645b6 with peer d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:23.213065 32203 raft_consensus.cc:1217] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.31->[1.32-1.32] Dedup: 1.32->[]
I20251027 03:09:23.263048 32052 raft_consensus.cc:1217] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20251027 03:09:23.396904 32203 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Advancing to term 3
I20251027 03:09:23.397845 32203 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 3 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:23.398038 32406 consensus_queue.cc:1050] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:23.398830 32406 raft_consensus.cc:1064] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: attempting to promote NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d to VOTER
I20251027 03:09:23.398985 32406 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:23.399335 32203 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 3 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:23.399354 31717 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:23.399354 31639 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:23.399526 32406 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:23.399627 32406 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:23.399718 32406 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:23.400179 32389 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.400483 31639 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.401325 32203 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.401419 31539 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 28 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:23.401733 31717 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.403389 31792 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:23.403712 31639 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:23.403716 32203 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:23.403883 32314 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:23.403962 32406 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:23.404484 32406 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.404695 31639 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.405272 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 914edf6a5ed5402fb1d07b6afba645b6 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:23.404695 32203 raft_consensus.cc:2955] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.405452 31540 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 31 to 32, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:23.406689 31704 tablet_service.cc:1552] Processing DeleteTablet for tablet 914edf6a5ed5402fb1d07b6afba645b6 with delete_type TABLET_DATA_TOMBSTONED (TS 50aa97f11c164e0b9e3520267cf6c85f not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:23.406852 32417 tablet_replica.cc:333] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:23.406917 32417 raft_consensus.cc:2243] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:23.407016 32417 raft_consensus.cc:2272] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:23.407331 32417 ts_tablet_manager.cc:1916] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:23.408766 32417 ts_tablet_manager.cc:1929] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20251027 03:09:23.408855 32417 log.cc:1199] T 914edf6a5ed5402fb1d07b6afba645b6 P 50aa97f11c164e0b9e3520267cf6c85f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/914edf6a5ed5402fb1d07b6afba645b6
I20251027 03:09:23.409173 31529 catalog_manager.cc:4985] TS 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091): tablet 914edf6a5ed5402fb1d07b6afba645b6 (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:23.781499 32418 raft_consensus.cc:1064] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d to VOTER
I20251027 03:09:23.781780 32418 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:23.782342 32203 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251027 03:09:23.782363 32052 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251027 03:09:23.782342 31792 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20251027 03:09:23.782658 32379 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:23.782742 32421 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:23.782799 32420 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:23.783624 32379 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.784003 32052 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.784891 31539 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 32 to 33, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:23.785408 31792 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.785703 31640 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:23.785995 32052 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20251027 03:09:23.786266 32420 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:23.786550 32203 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.786998 32420 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.787318 32202 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 1 index: 33. Preceding OpId from leader: term: 1 index: 34. (index mismatch)
I20251027 03:09:23.787604 32379 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:23.787945 32202 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.788117 31540 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 33 to 34, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New cstate: current_term: 1 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:23.788228 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6045341ee60c472bbb01e952650a735a with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:23.788841 31779 tablet_service.cc:1552] Processing DeleteTablet for tablet 6045341ee60c472bbb01e952650a735a with delete_type TABLET_DATA_TOMBSTONED (TS 10153e76038242fdaec0d401cdbb0411 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:52962
I20251027 03:09:23.789021 32052 raft_consensus.cc:2955] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.34: config changed from index 33 to 34, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:23.789335 32426 tablet_replica.cc:333] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:23.789397 32426 raft_consensus.cc:2243] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:23.789484 32426 raft_consensus.cc:2272] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:23.789706 32426 ts_tablet_manager.cc:1916] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:23.791050 32426 ts_tablet_manager.cc:1929] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.33
I20251027 03:09:23.791106 32426 log.cc:1199] T 6045341ee60c472bbb01e952650a735a P 10153e76038242fdaec0d401cdbb0411: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/6045341ee60c472bbb01e952650a735a
I20251027 03:09:23.791460 31527 catalog_manager.cc:4985] TS 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045): tablet 6045341ee60c472bbb01e952650a735a (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:23.872936 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:23.873363 31792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:23.873450 31792 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:23.873739 31792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:23.873826 31792 raft_consensus.cc:606] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:23.874118 31792 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:23.874173 31792 raft_consensus.cc:606] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:23.874437 31719 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:23.874495 31719 raft_consensus.cc:606] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:23.874617 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251027 03:09:23.874694 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:23.881083 32427 raft_consensus.cc:1064] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:23.881317 32427 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:23.881814 31792 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:23.881855 31640 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:23.882009 32381 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:23.882097 32052 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 LEARNER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251027 03:09:23.882095 32427 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:23.882284 32381 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251027 03:09:23.883701 32430 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.883790 31792 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.883790 31640 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.884977 31540 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 27 to 28, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.885408 32052 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.885761 31719 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:23.886118 31640 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:23.886217 32051 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:23.886268 32429 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:23.886545 32429 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:23.886869 32429 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.887303 32051 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.887967 31538 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index 28 to 29, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:23.888413 31529 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c176d9f872b84ac986522990949395eb with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:23.888418 31640 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:23.888872 31779 tablet_service.cc:1552] Processing DeleteTablet for tablet c176d9f872b84ac986522990949395eb with delete_type TABLET_DATA_TOMBSTONED (TS 10153e76038242fdaec0d401cdbb0411 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:52962
I20251027 03:09:23.888972 32426 tablet_replica.cc:333] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:23.889029 32426 raft_consensus.cc:2243] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:23.889086 32426 raft_consensus.cc:2272] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:23.889362 32426 ts_tablet_manager.cc:1916] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:23.890378 32426 ts_tablet_manager.cc:1929] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20251027 03:09:23.890426 32426 log.cc:1199] T c176d9f872b84ac986522990949395eb P 10153e76038242fdaec0d401cdbb0411: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/c176d9f872b84ac986522990949395eb
I20251027 03:09:23.890739 31527 catalog_manager.cc:4985] TS 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045): tablet c176d9f872b84ac986522990949395eb (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:24.015095 32314 raft_consensus.cc:993] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:24.015184 32314 raft_consensus.cc:1081] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:24.015378 31640 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43380
I20251027 03:09:24.015465 31640 raft_consensus.cc:493] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:24.015496 31640 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:24.016112 31640 raft_consensus.cc:515] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:24.016278 31640 leader_election.cc:290] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 4 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:24.016414 32202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
I20251027 03:09:24.016414 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:24.016507 32202 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:24.016515 31792 raft_consensus.cc:3055] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:24.016552 31792 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:24.016633 31792 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:24.016721 31792 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:24.017338 32202 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 4.
I20251027 03:09:24.017338 31792 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 4.
I20251027 03:09:24.017535 31611 leader_election.cc:304] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78e34a95a4a44fde91fa13a71fa3da26, d421b303d20a48cfbbd34da50b8d0d6d; no voters:
I20251027 03:09:24.017673 32379 raft_consensus.cc:2804] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:24.017738 32379 raft_consensus.cc:697] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:24.017800 32379 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:24.018254 31539 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 3 to 4, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 4 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:24.227180 32430 raft_consensus.cc:993] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:24.227264 32430 raft_consensus.cc:1081] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:24.227445 31792 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:24.227536 31792 raft_consensus.cc:493] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:24.227558 31792 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:24.228179 31792 raft_consensus.cc:515] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:24.228299 31792 leader_election.cc:290] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:24.228411 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:24.228473 31640 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:24.228459 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:24.228534 31717 raft_consensus.cc:3055] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:24.228564 31717 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:24.228619 31717 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:24.228693 31717 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:24.229199 31640 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:24.229292 31717 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:24.229368 31760 leader_election.cc:304] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:24.229447 32314 raft_consensus.cc:2804] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:24.229503 32314 raft_consensus.cc:697] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:24.229573 32314 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:24.230118 31539 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 2 to 3, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:24.295466 32314 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:24.295533 32314 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:24.295692 31640 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43380
I20251027 03:09:24.295749 31640 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:24.295768 31640 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:24.296386 31640 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:24.296504 31640 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:24.296674 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:24.296698 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:24.296761 31792 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:24.296768 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:24.296803 31792 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:24.296867 31792 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:24.296941 31792 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:24.297344 31717 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:24.297478 31792 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:24.297497 31611 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:24.297564 32379 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:24.297605 32379 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:24.297652 32379 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:24.298272 31539 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 2 to 3, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:24.352820 32314 raft_consensus.cc:993] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:24.352908 32314 raft_consensus.cc:1081] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:24.353070 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57726
I20251027 03:09:24.353127 31717 raft_consensus.cc:493] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:24.353145 31717 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:24.353675 31717 raft_consensus.cc:515] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:24.353802 31717 leader_election.cc:290] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 2 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233)
I20251027 03:09:24.353953 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:24.354048 31792 raft_consensus.cc:3055] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:24.354074 31792 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:24.354127 31792 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:24.354195 31792 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:24.354777 31792 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 2.
I20251027 03:09:24.354943 31685 leader_election.cc:304] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:24.355028 32430 raft_consensus.cc:2804] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:24.355091 32430 raft_consensus.cc:697] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:24.355139 32430 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:24.355629 31539 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 1 to 2, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 2 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:24.356817 32128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
I20251027 03:09:24.356866 32128 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:24.357334 32128 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 2.
I20251027 03:09:24.569797 31792 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:24.570039 32379 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:24.571106 32202 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:24.571273 32442 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:24.720681 31792 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:24.720917 32379 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:24.721896 31717 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:24.722112 32442 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:24.733831 32128 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 2 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251027 03:09:24.734026 32430 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:24.734969 31792 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251027 03:09:24.735122 32430 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:24.827307 31640 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:24.827587 32314 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:24.828567 31717 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:24.828727 32454 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:24.895596 32202 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:24.896204 31640 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20251027 03:09:24.896212 31717 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20251027 03:09:24.896445 32413 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:24.896556 32459 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:24.897265 32459 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.897408 31717 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.897423 31640 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.898474 31539 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 31 to 33, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New cstate: current_term: 2 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.899120 31717 consensus_queue.cc:237] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:24.899608 31640 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251027 03:09:24.899631 31792 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20251027 03:09:24.899780 31686 consensus_peers.cc:597] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb4aebbafbe6415e90eaf6f6feb6b760. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:24.899976 32430 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:24.900107 32451 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:24.900843 32381 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:24.900944 31792 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:24.901094 31640 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:24.901881 31540 catalog_manager.cc:5649] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index -1 to 29, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
W20251027 03:09:24.902046 32169 consensus_peers.cc:597] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d -> Peer 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331): Couldn't send request to peer 93e976cbba5e4213a43042ffe2da42d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 645f3c62551e42109ebfe785bde36416. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:24.902251 31792 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:24.902750 31717 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
W20251027 03:09:24.902787 31762 consensus_peers.cc:597] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1ec5a8f7224329be0fe0e5453d71fc. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:24.902792 31640 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251027 03:09:24.902941 32454 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:24.903054 32314 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251027 03:09:24.903542 32454 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:24.903657 31717 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:24.903716 31640 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:24.904593 31539 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index -1 to 29, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New cstate: current_term: 2 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:24.904925 31640 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
W20251027 03:09:24.905493 31610 consensus_peers.cc:597] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30ef3ff09aa247fab2eae47a3f1718fd. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:24.905579 31717 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:24.905583 31792 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:24.905730 32421 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:24.905817 32442 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:24.906435 32442 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index -1 to 31, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:24.906564 31717 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index -1 to 31, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:24.906564 31792 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index -1 to 31, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:24.907295 31540 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index -1 to 31, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:24.907884 31792 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:24.908394 31640 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251027 03:09:24.908524 31717 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251027 03:09:24.908596 32434 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:24.908696 32454 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:24.909375 32454 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.910182 31717 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.910228 31539 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index -1 to 30, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New cstate: current_term: 2 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:24.910548 31792 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:24.910914 31640 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
W20251027 03:09:24.911955 31763 consensus_peers.cc:597] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 -> Peer 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331): Couldn't send request to peer 93e976cbba5e4213a43042ffe2da42d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a4ca641cf349f29eeb0292a996449a. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:24.911955 31640 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
W20251027 03:09:24.912035 31763 consensus_peers.cc:597] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 -> Peer 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331): Couldn't send request to peer 93e976cbba5e4213a43042ffe2da42d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11ce7fc7b4334ffe99df2944dca2af4a. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:24.912072 31717 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:24.912217 32454 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:24.912271 32434 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:24.912874 32454 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index -1 to 31, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.912981 31640 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index -1 to 31, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.913028 31717 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index -1 to 31, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:24.913911 31538 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index -1 to 31, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:25.366688 32482 ts_tablet_manager.cc:933] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: Initiating tablet copy from peer 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:25.366873 32482 tablet_copy_client.cc:323] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:41091
I20251027 03:09:25.367079 31732 tablet_copy_service.cc:140] P 50aa97f11c164e0b9e3520267cf6c85f: Received BeginTabletCopySession request for tablet bb4aebbafbe6415e90eaf6f6feb6b760 from peer bcd3eff6ab6f42e9b6b475710ba14c80 ({username='slave'} at 127.0.0.1:57758)
I20251027 03:09:25.367143 31732 tablet_copy_service.cc:161] P 50aa97f11c164e0b9e3520267cf6c85f: Beginning new tablet copy session on tablet bb4aebbafbe6415e90eaf6f6feb6b760 from peer bcd3eff6ab6f42e9b6b475710ba14c80 at {username='slave'} at 127.0.0.1:57758: session id = bcd3eff6ab6f42e9b6b475710ba14c80-bb4aebbafbe6415e90eaf6f6feb6b760
I20251027 03:09:25.367588 31732 tablet_copy_source_session.cc:215] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:25.367913 32482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb4aebbafbe6415e90eaf6f6feb6b760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:25.369122 32482 tablet_copy_client.cc:806] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:25.369263 32482 tablet_copy_client.cc:670] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:25.369782 32483 tablet_copy_client.cc:1025] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:25.400352 32484 ts_tablet_manager.cc:933] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Initiating tablet copy from peer 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:25.400487 32484 tablet_copy_client.cc:323] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:37045
I20251027 03:09:25.403178 31807 tablet_copy_service.cc:140] P 10153e76038242fdaec0d401cdbb0411: Received BeginTabletCopySession request for tablet 5e1ec5a8f7224329be0fe0e5453d71fc from peer d421b303d20a48cfbbd34da50b8d0d6d ({username='slave'} at 127.0.0.1:49700)
I20251027 03:09:25.403234 31807 tablet_copy_service.cc:161] P 10153e76038242fdaec0d401cdbb0411: Beginning new tablet copy session on tablet 5e1ec5a8f7224329be0fe0e5453d71fc from peer d421b303d20a48cfbbd34da50b8d0d6d at {username='slave'} at 127.0.0.1:49700: session id = d421b303d20a48cfbbd34da50b8d0d6d-5e1ec5a8f7224329be0fe0e5453d71fc
I20251027 03:09:25.403681 31807 tablet_copy_source_session.cc:215] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:25.403942 32484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e1ec5a8f7224329be0fe0e5453d71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:25.404953 32484 tablet_copy_client.cc:806] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:25.405053 32484 tablet_copy_client.cc:670] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:25.405496 32485 tablet_copy_client.cc:1025] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:25.409758 32487 ts_tablet_manager.cc:933] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: Initiating tablet copy from peer 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:25.409883 32487 tablet_copy_client.cc:323] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:37045
I20251027 03:09:25.412536 31807 tablet_copy_service.cc:140] P 10153e76038242fdaec0d401cdbb0411: Received BeginTabletCopySession request for tablet a7a4ca641cf349f29eeb0292a996449a from peer 93e976cbba5e4213a43042ffe2da42d6 ({username='slave'} at 127.0.0.1:49702)
I20251027 03:09:25.412583 31807 tablet_copy_service.cc:161] P 10153e76038242fdaec0d401cdbb0411: Beginning new tablet copy session on tablet a7a4ca641cf349f29eeb0292a996449a from peer 93e976cbba5e4213a43042ffe2da42d6 at {username='slave'} at 127.0.0.1:49702: session id = 93e976cbba5e4213a43042ffe2da42d6-a7a4ca641cf349f29eeb0292a996449a
I20251027 03:09:25.412945 31807 tablet_copy_source_session.cc:215] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:25.413175 32487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a4ca641cf349f29eeb0292a996449a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:25.414315 32487 tablet_copy_client.cc:806] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:25.414463 32487 tablet_copy_client.cc:670] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:25.414978 32488 tablet_copy_client.cc:1025] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:25.454388 32490 ts_tablet_manager.cc:933] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: Initiating tablet copy from peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:25.454519 32490 tablet_copy_client.cc:323] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.70:42021
I20251027 03:09:25.457172 32213 tablet_copy_service.cc:140] P d421b303d20a48cfbbd34da50b8d0d6d: Received BeginTabletCopySession request for tablet 645f3c62551e42109ebfe785bde36416 from peer 93e976cbba5e4213a43042ffe2da42d6 ({username='slave'} at 127.0.0.1:47842)
I20251027 03:09:25.457224 32213 tablet_copy_service.cc:161] P d421b303d20a48cfbbd34da50b8d0d6d: Beginning new tablet copy session on tablet 645f3c62551e42109ebfe785bde36416 from peer 93e976cbba5e4213a43042ffe2da42d6 at {username='slave'} at 127.0.0.1:47842: session id = 93e976cbba5e4213a43042ffe2da42d6-645f3c62551e42109ebfe785bde36416
I20251027 03:09:25.457588 32213 tablet_copy_source_session.cc:215] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:25.457840 32490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 645f3c62551e42109ebfe785bde36416. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:25.459051 32490 tablet_copy_client.cc:806] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:25.459154 32490 tablet_copy_client.cc:670] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:25.459573 32491 tablet_copy_client.cc:1025] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:25.495712 32493 ts_tablet_manager.cc:933] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: Initiating tablet copy from peer 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:25.495867 32493 tablet_copy_client.cc:323] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:37045
I20251027 03:09:25.496024 31807 tablet_copy_service.cc:140] P 10153e76038242fdaec0d401cdbb0411: Received BeginTabletCopySession request for tablet 11ce7fc7b4334ffe99df2944dca2af4a from peer 93e976cbba5e4213a43042ffe2da42d6 ({username='slave'} at 127.0.0.1:49702)
I20251027 03:09:25.496083 31807 tablet_copy_service.cc:161] P 10153e76038242fdaec0d401cdbb0411: Beginning new tablet copy session on tablet 11ce7fc7b4334ffe99df2944dca2af4a from peer 93e976cbba5e4213a43042ffe2da42d6 at {username='slave'} at 127.0.0.1:49702: session id = 93e976cbba5e4213a43042ffe2da42d6-11ce7fc7b4334ffe99df2944dca2af4a
I20251027 03:09:25.496577 31807 tablet_copy_source_session.cc:215] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:25.496827 32493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ce7fc7b4334ffe99df2944dca2af4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:25.497984 32493 tablet_copy_client.cc:806] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:25.498090 32493 tablet_copy_client.cc:670] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:25.498540 32494 tablet_copy_client.cc:1025] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:25.519776 32495 ts_tablet_manager.cc:933] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: Initiating tablet copy from peer 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:25.519941 32495 tablet_copy_client.cc:323] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:32889
I20251027 03:09:25.522778 31656 tablet_copy_service.cc:140] P 78e34a95a4a44fde91fa13a71fa3da26: Received BeginTabletCopySession request for tablet 30ef3ff09aa247fab2eae47a3f1718fd from peer bcd3eff6ab6f42e9b6b475710ba14c80 ({username='slave'} at 127.0.0.1:48608)
I20251027 03:09:25.522835 31656 tablet_copy_service.cc:161] P 78e34a95a4a44fde91fa13a71fa3da26: Beginning new tablet copy session on tablet 30ef3ff09aa247fab2eae47a3f1718fd from peer bcd3eff6ab6f42e9b6b475710ba14c80 at {username='slave'} at 127.0.0.1:48608: session id = bcd3eff6ab6f42e9b6b475710ba14c80-30ef3ff09aa247fab2eae47a3f1718fd
I20251027 03:09:25.523190 31656 tablet_copy_source_session.cc:215] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:25.523442 32495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ef3ff09aa247fab2eae47a3f1718fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:25.524257 32495 tablet_copy_client.cc:806] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:25.524348 32495 tablet_copy_client.cc:670] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:25.524981 32496 tablet_copy_client.cc:1025] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:25.874840 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:25.875263 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:25.875341 31640 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:25.875609 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:25.875672 31795 raft_consensus.cc:606] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:25.875991 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:25.876065 31717 raft_consensus.cc:606] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:25.876368 31640 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:25.876434 31640 raft_consensus.cc:606] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:25.876565 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251027 03:09:25.876632 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:26.051270 32501 raft_consensus.cc:993] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:26.051362 32501 raft_consensus.cc:1081] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:26.051507 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57726
I20251027 03:09:26.051558 32503 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:26.051592 31717 raft_consensus.cc:493] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:26.051615 32503 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:26.051623 31717 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:26.052443 31717 raft_consensus.cc:515] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:26.052573 31717 leader_election.cc:290] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 3 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:26.052687 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:26.052755 31639 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:26.052838 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:26.052894 31717 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:26.052915 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:26.053390 31717 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:26.053517 31717 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 4 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:26.053390 31639 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 3.
I20251027 03:09:26.053706 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:26.053784 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:26.053819 31685 leader_election.cc:304] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:26.053900 32471 raft_consensus.cc:2804] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:26.053941 32471 raft_consensus.cc:697] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:26.053993 32471 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:26.054567 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 4.
I20251027 03:09:26.053673 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:26.054580 31540 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 2 to 3, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 3 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:26.053417 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:26.054685 31640 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:26.054742 31792 raft_consensus.cc:3055] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:26.054751 31640 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:26.054816 31792 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:26.054862 31640 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:26.054913 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:26.054946 31640 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:26.054979 32499 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:26.055015 32499 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:26.055060 32499 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:26.055655 31640 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 4.
I20251027 03:09:26.054869 31792 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:26.055950 31792 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:26.056536 31540 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 3 to 4, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 4 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:26.056707 31792 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 3.
I20251027 03:09:26.057394 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:26.057451 31717 raft_consensus.cc:606] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Received request to transfer leadership
I20251027 03:09:26.059013 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:26.059096 31717 raft_consensus.cc:606] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Received request to transfer leadership
W20251027 03:09:26.059193 31529 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 5e1ec5a8f7224329be0fe0e5453d71fc already in progress [suppressed 1 similar messages]
I20251027 03:09:26.059465 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:26.059538 31717 raft_consensus.cc:606] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Received request to transfer leadership
W20251027 03:09:26.059676 31592 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 5e1ec5a8f7224329be0fe0e5453d71fc already in progress
W20251027 03:09:26.059718 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 5e1ec5a8f7224329be0fe0e5453d71fc already in progress
I20251027 03:09:26.142547 32503 raft_consensus.cc:993] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:26.142638 32503 raft_consensus.cc:1081] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:26.142771 31795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52978
I20251027 03:09:26.142843 31795 raft_consensus.cc:493] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:26.142882 31795 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:26.143529 31795 raft_consensus.cc:515] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:26.143712 31795 leader_election.cc:290] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 5 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:26.143812 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:26.143812 32202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "914edf6a5ed5402fb1d07b6afba645b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
I20251027 03:09:26.143916 31640 raft_consensus.cc:3055] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 LEADER]: Stepping down as leader of term 4
I20251027 03:09:26.143925 32202 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:26.143956 31640 raft_consensus.cc:740] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:26.144009 31640 consensus_queue.cc:260] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:26.144101 31640 raft_consensus.cc:3060] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:26.144642 32202 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 5.
I20251027 03:09:26.144755 31640 raft_consensus.cc:2468] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 5.
I20251027 03:09:26.144913 31760 leader_election.cc:304] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:26.145056 32434 raft_consensus.cc:2804] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Leader election won for term 5
I20251027 03:09:26.145107 32434 raft_consensus.cc:697] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:26.145156 32434 consensus_queue.cc:237] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:26.145637 31540 catalog_manager.cc:5649] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 4 to 5, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 5 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:26.191056 32471 raft_consensus.cc:993] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:26.191138 32471 raft_consensus.cc:1081] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:26.191272 31795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:26.191341 31795 raft_consensus.cc:493] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:26.191366 31795 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:26.192063 31795 raft_consensus.cc:515] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:26.192205 31795 leader_election.cc:290] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233)
I20251027 03:09:26.192292 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:26.192310 32128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
I20251027 03:09:26.192353 31717 raft_consensus.cc:3055] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:26.192389 32128 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:26.192409 31717 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:26.192466 31717 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:26.192543 31717 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:26.193094 32128 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:26.193094 31717 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 3.
I20251027 03:09:26.193269 31762 leader_election.cc:304] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:26.193361 32434 raft_consensus.cc:2804] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:26.193403 32434 raft_consensus.cc:697] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:26.193440 32434 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:26.193905 31540 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 2 to 3, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20251027 03:09:26.527014 31687 consensus_peers.cc:597] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:26.581681 31792 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20251027 03:09:26.581912 32499 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:26.582928 31639 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20251027 03:09:26.583118 32499 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:26.612614 31792 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:26.612813 32471 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:26.613443 32499 raft_consensus.cc:993] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:26.613502 32499 raft_consensus.cc:1081] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:26.613716 31795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:26.613790 31795 raft_consensus.cc:493] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:26.613816 31795 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:26.613819 31639 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:26.614018 32520 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:26.614537 31795 raft_consensus.cc:515] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:26.614677 31795 leader_election.cc:290] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 4 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:26.614765 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:26.614846 31639 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:26.615558 31639 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 4.
I20251027 03:09:26.614846 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:26.615692 31717 raft_consensus.cc:3055] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:26.615728 31717 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:26.615780 31717 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:26.615871 31717 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:26.615695 31760 leader_election.cc:304] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:26.616354 32501 raft_consensus.cc:2804] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:26.616431 32501 raft_consensus.cc:697] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:26.616487 32501 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:26.616535 31717 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 4.
I20251027 03:09:26.617112 31540 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 3 to 4, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 4 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:26.659017 31639 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20251027 03:09:26.659193 32501 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:26.660084 32202 raft_consensus.cc:1275] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20251027 03:09:26.660254 32501 consensus_queue.cc:1048] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:26.698735 31719 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:26.698941 32501 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:26.699854 32128 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:26.700274 32500 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:27.022163 31639 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20251027 03:09:27.022380 32501 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:27.023218 31719 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
W20251027 03:09:27.023329 31762 consensus_peers.cc:597] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:27.023497 32434 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:27.369968 32482 tablet_copy_client.cc:538] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:27.370744 32482 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap starting.
I20251027 03:09:27.374490 32482 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:27.374737 32482 tablet_bootstrap.cc:492] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap complete.
I20251027 03:09:27.374848 32482 ts_tablet_manager.cc:1403] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.002s
I20251027 03:09:27.375029 32482 raft_consensus.cc:359] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:27.375116 32482 raft_consensus.cc:740] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Initialized, Role: LEARNER
I20251027 03:09:27.375150 32482 consensus_queue.cc:260] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:27.375280 32482 ts_tablet_manager.cc:1434] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:09:27.375490 31732 tablet_copy_service.cc:342] P 50aa97f11c164e0b9e3520267cf6c85f: Request end of tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-bb4aebbafbe6415e90eaf6f6feb6b760 received from {username='slave'} at 127.0.0.1:57758
I20251027 03:09:27.375530 31732 tablet_copy_service.cc:434] P 50aa97f11c164e0b9e3520267cf6c85f: ending tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-bb4aebbafbe6415e90eaf6f6feb6b760 on tablet bb4aebbafbe6415e90eaf6f6feb6b760 with peer bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:27.405673 32484 tablet_copy_client.cc:538] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:27.406322 32484 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap starting.
I20251027 03:09:27.410109 32484 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:27.410305 32484 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap complete.
I20251027 03:09:27.410409 32484 ts_tablet_manager.cc:1403] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251027 03:09:27.410560 32484 raft_consensus.cc:359] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:27.410655 32484 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Initialized, Role: LEARNER
I20251027 03:09:27.410699 32484 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:27.410840 32484 ts_tablet_manager.cc:1434] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20251027 03:09:27.411010 31807 tablet_copy_service.cc:342] P 10153e76038242fdaec0d401cdbb0411: Request end of tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-5e1ec5a8f7224329be0fe0e5453d71fc received from {username='slave'} at 127.0.0.1:49700
I20251027 03:09:27.411068 31807 tablet_copy_service.cc:434] P 10153e76038242fdaec0d401cdbb0411: ending tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-5e1ec5a8f7224329be0fe0e5453d71fc on tablet 5e1ec5a8f7224329be0fe0e5453d71fc with peer d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:27.415172 32487 tablet_copy_client.cc:538] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:27.415841 32487 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap starting.
I20251027 03:09:27.419014 32487 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=105 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:27.419240 32487 tablet_bootstrap.cc:492] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap complete.
I20251027 03:09:27.419348 32487 ts_tablet_manager.cc:1403] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20251027 03:09:27.419508 32487 raft_consensus.cc:359] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:27.419584 32487 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Initialized, Role: LEARNER
I20251027 03:09:27.419624 32487 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:27.419751 32487 ts_tablet_manager.cc:1434] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:09:27.419934 31807 tablet_copy_service.cc:342] P 10153e76038242fdaec0d401cdbb0411: Request end of tablet copy session 93e976cbba5e4213a43042ffe2da42d6-a7a4ca641cf349f29eeb0292a996449a received from {username='slave'} at 127.0.0.1:49702
I20251027 03:09:27.419979 31807 tablet_copy_service.cc:434] P 10153e76038242fdaec0d401cdbb0411: ending tablet copy session 93e976cbba5e4213a43042ffe2da42d6-a7a4ca641cf349f29eeb0292a996449a on tablet a7a4ca641cf349f29eeb0292a996449a with peer 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:27.459743 32490 tablet_copy_client.cc:538] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:27.460695 32490 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap starting.
I20251027 03:09:27.464476 32490 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:27.464705 32490 tablet_bootstrap.cc:492] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap complete.
I20251027 03:09:27.464823 32490 ts_tablet_manager.cc:1403] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251027 03:09:27.465009 32490 raft_consensus.cc:359] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:27.465087 32490 raft_consensus.cc:740] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Initialized, Role: LEARNER
I20251027 03:09:27.465137 32490 consensus_queue.cc:260] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:27.465235 32490 ts_tablet_manager.cc:1434] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:09:27.465399 32213 tablet_copy_service.cc:342] P d421b303d20a48cfbbd34da50b8d0d6d: Request end of tablet copy session 93e976cbba5e4213a43042ffe2da42d6-645f3c62551e42109ebfe785bde36416 received from {username='slave'} at 127.0.0.1:47842
I20251027 03:09:27.465459 32213 tablet_copy_service.cc:434] P d421b303d20a48cfbbd34da50b8d0d6d: ending tablet copy session 93e976cbba5e4213a43042ffe2da42d6-645f3c62551e42109ebfe785bde36416 on tablet 645f3c62551e42109ebfe785bde36416 with peer 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:27.474088 32052 raft_consensus.cc:1217] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.32->[2.33-2.33] Dedup: 2.33->[]
I20251027 03:09:27.498675 32493 tablet_copy_client.cc:538] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:27.499533 32493 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap starting.
I20251027 03:09:27.503612 32493 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:27.503808 32493 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap complete.
I20251027 03:09:27.503913 32493 ts_tablet_manager.cc:1403] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251027 03:09:27.504055 32493 raft_consensus.cc:359] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:27.504137 32493 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Initialized, Role: LEARNER
I20251027 03:09:27.504196 32493 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:27.504287 32493 ts_tablet_manager.cc:1434] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s
I20251027 03:09:27.504474 31807 tablet_copy_service.cc:342] P 10153e76038242fdaec0d401cdbb0411: Request end of tablet copy session 93e976cbba5e4213a43042ffe2da42d6-11ce7fc7b4334ffe99df2944dca2af4a received from {username='slave'} at 127.0.0.1:49702
I20251027 03:09:27.504537 31807 tablet_copy_service.cc:434] P 10153e76038242fdaec0d401cdbb0411: ending tablet copy session 93e976cbba5e4213a43042ffe2da42d6-11ce7fc7b4334ffe99df2944dca2af4a on tablet 11ce7fc7b4334ffe99df2944dca2af4a with peer 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:27.510210 32203 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEARNER]: Advancing to term 4
I20251027 03:09:27.510900 32203 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251027 03:09:27.511076 32500 consensus_queue.cc:1050] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:27.511909 32434 raft_consensus.cc:1064] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: attempting to promote NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d to VOTER
I20251027 03:09:27.512080 32434 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 30, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:27.512496 32203 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251027 03:09:27.512529 31719 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251027 03:09:27.512548 31639 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251027 03:09:27.512651 32538 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:27.512722 32500 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:27.512945 32500 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:27.513319 32538 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 29 to 32, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.513414 32202 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.514437 31639 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.514470 31540 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 29 to 32, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:27.515200 31717 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 29 to 32, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.515779 31795 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:27.516072 32202 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:27.516235 32500 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:27.516314 31639 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:27.516562 32500 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:27.516697 32501 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.516808 32202 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.517501 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 5e1ec5a8f7224329be0fe0e5453d71fc with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:27.517673 31538 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 32 to 33, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New cstate: current_term: 4 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:27.518126 31639 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:27.518378 31704 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e1ec5a8f7224329be0fe0e5453d71fc with delete_type TABLET_DATA_TOMBSTONED (TS 50aa97f11c164e0b9e3520267cf6c85f not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:27.518705 32542 tablet_replica.cc:333] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:27.518833 32542 raft_consensus.cc:2243] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:27.518940 32542 raft_consensus.cc:2272] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:27.519150 32542 ts_tablet_manager.cc:1916] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:27.520305 32542 ts_tablet_manager.cc:1929] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20251027 03:09:27.520359 32542 log.cc:1199] T 5e1ec5a8f7224329be0fe0e5453d71fc P 50aa97f11c164e0b9e3520267cf6c85f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/5e1ec5a8f7224329be0fe0e5453d71fc
I20251027 03:09:27.520713 31529 catalog_manager.cc:4985] TS 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091): tablet 5e1ec5a8f7224329be0fe0e5453d71fc (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:27.525144 32495 tablet_copy_client.cc:538] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:27.525635 32495 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap starting.
I20251027 03:09:27.528715 32495 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=133 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:27.528890 32495 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap complete.
I20251027 03:09:27.528988 32495 ts_tablet_manager.cc:1403] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20251027 03:09:27.529197 32495 raft_consensus.cc:359] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:27.529247 32495 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Initialized, Role: LEARNER
I20251027 03:09:27.529280 32495 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:27.529412 32495 ts_tablet_manager.cc:1434] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:09:27.529582 31656 tablet_copy_service.cc:342] P 78e34a95a4a44fde91fa13a71fa3da26: Request end of tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-30ef3ff09aa247fab2eae47a3f1718fd received from {username='slave'} at 127.0.0.1:48608
I20251027 03:09:27.529649 31656 tablet_copy_service.cc:434] P 78e34a95a4a44fde91fa13a71fa3da26: ending tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-30ef3ff09aa247fab2eae47a3f1718fd on tablet 30ef3ff09aa247fab2eae47a3f1718fd with peer bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:27.561184 32128 raft_consensus.cc:1217] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20251027 03:09:27.600054 32052 raft_consensus.cc:1217] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20251027 03:09:27.636276 32128 raft_consensus.cc:1217] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20251027 03:09:27.822348 32052 raft_consensus.cc:1217] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20251027 03:09:27.850833 32522 raft_consensus.cc:1064] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:27.851145 32522 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 31, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:27.851696 31640 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20251027 03:09:27.851814 31717 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20251027 03:09:27.851909 32052 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20251027 03:09:27.851890 32546 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:27.851976 32537 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:27.852281 32537 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:27.853080 32537 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.853209 31640 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.854270 31717 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.854285 31540 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: config changed from index 33 to 34, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:27.855225 32052 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.855429 32202 consensus_queue.cc:237] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:27.856024 32051 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20251027 03:09:27.856272 31640 raft_consensus.cc:1275] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20251027 03:09:27.856503 32546 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:27.856578 32537 consensus_queue.cc:1048] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:27.857088 32537 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.857268 31640 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.857863 31590 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 645f3c62551e42109ebfe785bde36416 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:27.857934 32051 raft_consensus.cc:2955] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:27.858063 31540 catalog_manager.cc:5649] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: config changed from index 34 to 35, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New cstate: current_term: 2 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:27.858770 31704 tablet_service.cc:1552] Processing DeleteTablet for tablet 645f3c62551e42109ebfe785bde36416 with delete_type TABLET_DATA_TOMBSTONED (TS 50aa97f11c164e0b9e3520267cf6c85f not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:27.858875 32542 tablet_replica.cc:333] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:27.858935 32542 raft_consensus.cc:2243] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:27.859016 32542 raft_consensus.cc:2272] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:27.859272 32542 ts_tablet_manager.cc:1916] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:27.860661 32542 ts_tablet_manager.cc:1929] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20251027 03:09:27.860719 32542 log.cc:1199] T 645f3c62551e42109ebfe785bde36416 P 50aa97f11c164e0b9e3520267cf6c85f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/645f3c62551e42109ebfe785bde36416
I20251027 03:09:27.861057 31529 catalog_manager.cc:4985] TS 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091): tablet 645f3c62551e42109ebfe785bde36416 (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:27.876757 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:27.877123 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:27.877194 31795 raft_consensus.cc:606] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:27.877485 31719 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:27.877560 31719 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:27.877775 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:27.877835 31795 raft_consensus.cc:606] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Received request to transfer leadership to TS d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:27.878078 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:27.878150 31795 raft_consensus.cc:606] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:27.878268 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20251027 03:09:27.878327 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:27.891779 32500 raft_consensus.cc:993] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:27.891839 32500 raft_consensus.cc:1081] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:27.891991 31719 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57726
I20251027 03:09:27.892055 31719 raft_consensus.cc:493] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:27.892089 31719 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:27.892800 31719 raft_consensus.cc:515] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:27.892923 31719 leader_election.cc:290] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 4 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233)
I20251027 03:09:27.893112 32128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
I20251027 03:09:27.893113 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:27.893203 32128 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:27.893222 31792 raft_consensus.cc:3055] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:27.893260 31792 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:27.893316 31792 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:27.893414 31792 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:27.893857 32128 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 4.
I20251027 03:09:27.893975 31686 leader_election.cc:304] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, bcd3eff6ab6f42e9b6b475710ba14c80; no voters:
I20251027 03:09:27.894008 31792 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 4.
I20251027 03:09:27.894030 32499 raft_consensus.cc:2804] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:27.894109 32499 raft_consensus.cc:697] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:27.894155 32499 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:27.894816 31538 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 3 to 4, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 4 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:28.044248 32538 raft_consensus.cc:993] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: : Instructing follower d421b303d20a48cfbbd34da50b8d0d6d to start an election
I20251027 03:09:28.044354 32538 raft_consensus.cc:1081] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Signalling peer d421b303d20a48cfbbd34da50b8d0d6d to start an election
I20251027 03:09:28.044529 32202 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc"
dest_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
from {username='slave'} at 127.0.0.1:47830
I20251027 03:09:28.044610 32202 raft_consensus.cc:493] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:28.044639 32202 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:28.045486 32202 raft_consensus.cc:515] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:28.045662 32202 leader_election.cc:290] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [CANDIDATE]: Term 5 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:28.045758 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:28.045809 31640 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:28.045779 31792 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e1ec5a8f7224329be0fe0e5453d71fc" candidate_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:28.045887 31792 raft_consensus.cc:3055] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Stepping down as leader of term 4
I20251027 03:09:28.045914 31792 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:28.045971 31792 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:28.046072 31792 raft_consensus.cc:3060] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:28.046551 31640 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d421b303d20a48cfbbd34da50b8d0d6d in term 5.
I20251027 03:09:28.046788 31792 raft_consensus.cc:2468] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d421b303d20a48cfbbd34da50b8d0d6d in term 5.
I20251027 03:09:28.046964 32166 leader_election.cc:304] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, d421b303d20a48cfbbd34da50b8d0d6d; no voters:
I20251027 03:09:28.047101 32546 raft_consensus.cc:2804] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Leader election won for term 5
I20251027 03:09:28.047164 32546 raft_consensus.cc:697] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 LEADER]: Becoming Leader. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Running, Role: LEADER
I20251027 03:09:28.047224 32546 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:28.047642 31538 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: term changed from 4 to 5, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70). New cstate: current_term: 5 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:28.107354 32500 raft_consensus.cc:993] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:28.107446 32500 raft_consensus.cc:1081] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:28.107654 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43380
I20251027 03:09:28.107750 31639 raft_consensus.cc:493] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:28.107782 31639 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:28.108822 31639 raft_consensus.cc:515] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:28.108999 31639 leader_election.cc:290] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:28.109133 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:28.109221 31719 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:28.109934 31719 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:28.109170 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:28.110086 31611 leader_election.cc:304] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:28.110100 31795 raft_consensus.cc:3055] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:28.110200 31795 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:28.110253 31795 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:28.110354 31795 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:28.110816 31795 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:28.110930 32502 raft_consensus.cc:2804] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:28.111017 32502 raft_consensus.cc:697] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:28.111069 32502 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:28.111559 31538 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 2 to 3, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:28.112946 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:28.113008 31639 raft_consensus.cc:606] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Received request to transfer leadership
I20251027 03:09:28.114086 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:28.114142 31639 raft_consensus.cc:606] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Received request to transfer leadership
I20251027 03:09:28.114468 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:28.114518 31639 raft_consensus.cc:606] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Received request to transfer leadership
W20251027 03:09:28.114693 31592 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 11ce7fc7b4334ffe99df2944dca2af4a already in progress
W20251027 03:09:28.114723 31592 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 11ce7fc7b4334ffe99df2944dca2af4a already in progress
I20251027 03:09:28.176893 32555 raft_consensus.cc:1064] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
I20251027 03:09:28.177110 32555 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:28.177825 31795 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:28.177800 32128 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:28.177958 31719 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:28.178020 32555 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.178128 32552 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.178205 32502 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.178934 31795 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.178822 32555 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.179865 31538 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 31 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.179895 32128 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.181079 31719 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.181434 31639 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:28.181895 31717 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251027 03:09:28.182042 32555 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.182166 32128 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251027 03:09:28.182327 32502 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.182839 32555 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.183019 32128 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.183131 31717 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.183945 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 30ef3ff09aa247fab2eae47a3f1718fd with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:28.184042 31540 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 32 to 33, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:28.184801 31779 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ef3ff09aa247fab2eae47a3f1718fd with delete_type TABLET_DATA_TOMBSTONED (TS 10153e76038242fdaec0d401cdbb0411 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:52962
I20251027 03:09:28.184976 32560 tablet_replica.cc:333] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:28.185045 32560 raft_consensus.cc:2243] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:28.185102 32560 raft_consensus.cc:2272] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:28.185343 32560 ts_tablet_manager.cc:1916] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:28.186568 32560 ts_tablet_manager.cc:1929] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20251027 03:09:28.186618 32560 log.cc:1199] T 30ef3ff09aa247fab2eae47a3f1718fd P 10153e76038242fdaec0d401cdbb0411: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/30ef3ff09aa247fab2eae47a3f1718fd
I20251027 03:09:28.186939 31527 catalog_manager.cc:4985] TS 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045): tablet 30ef3ff09aa247fab2eae47a3f1718fd (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:28.198037 32538 raft_consensus.cc:1064] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:28.198194 32538 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:28.198645 31719 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:28.198734 32051 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 LEARNER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:28.198740 31639 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251027 03:09:28.198815 32563 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.198952 32561 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.199110 32500 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.199846 32561 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.200034 31639 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.200744 31719 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.200973 32051 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.201603 31538 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 31 to 32, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:28.202484 31795 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:28.202788 31719 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251027 03:09:28.202847 32051 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251027 03:09:28.202989 32538 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.203070 32500 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.203457 32550 raft_consensus.cc:1064] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
I20251027 03:09:28.203605 32550 consensus_queue.cc:237] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:28.204092 31639 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:28.204092 32128 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 LEARNER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:28.204209 31792 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:28.204334 32499 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:28.204314 32563 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.204391 32568 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:28.204555 32499 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:28.205034 32051 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.205090 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet a7a4ca641cf349f29eeb0292a996449a with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:28.205791 31717 raft_consensus.cc:2955] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.205921 31540 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: config changed from index 32 to 33, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New cstate: current_term: 3 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.206444 31635 tablet_service.cc:1552] Processing DeleteTablet for tablet a7a4ca641cf349f29eeb0292a996449a with delete_type TABLET_DATA_TOMBSTONED (TS 78e34a95a4a44fde91fa13a71fa3da26 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:28.206661 32573 tablet_replica.cc:333] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:28.206612 32569 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.206712 32573 raft_consensus.cc:2243] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:28.206763 32573 raft_consensus.cc:2272] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:28.206945 32573 ts_tablet_manager.cc:1916] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:28.207427 32128 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.207726 31639 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.208452 31795 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.208597 31540 catalog_manager.cc:5649] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 reported cstate change: config changed from index 29 to 30, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.209986 31717 consensus_queue.cc:237] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:28.210321 31795 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:28.210305 32128 raft_consensus.cc:1275] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:28.210482 32550 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:28.210646 32550 consensus_queue.cc:1048] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:28.210918 32499 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.211020 31795 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.211318 32128 raft_consensus.cc:2955] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.211750 31529 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet bb4aebbafbe6415e90eaf6f6feb6b760 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:28.211889 31538 catalog_manager.cc:5649] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 30 to 31, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:28.212347 32573 ts_tablet_manager.cc:1929] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20251027 03:09:28.212437 32573 log.cc:1199] T a7a4ca641cf349f29eeb0292a996449a P 78e34a95a4a44fde91fa13a71fa3da26: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/a7a4ca641cf349f29eeb0292a996449a
I20251027 03:09:28.212535 31635 tablet_service.cc:1552] Processing DeleteTablet for tablet bb4aebbafbe6415e90eaf6f6feb6b760 with delete_type TABLET_DATA_TOMBSTONED (TS 78e34a95a4a44fde91fa13a71fa3da26 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:28.212792 32573 tablet_replica.cc:333] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:28.212852 32573 raft_consensus.cc:2243] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:28.212854 31527 catalog_manager.cc:4985] TS 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889): tablet a7a4ca641cf349f29eeb0292a996449a (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:28.212919 32573 raft_consensus.cc:2272] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:28.213080 32573 ts_tablet_manager.cc:1916] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:28.213857 32573 ts_tablet_manager.cc:1929] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20251027 03:09:28.213899 32573 log.cc:1199] T bb4aebbafbe6415e90eaf6f6feb6b760 P 78e34a95a4a44fde91fa13a71fa3da26: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/bb4aebbafbe6415e90eaf6f6feb6b760
I20251027 03:09:28.214171 31527 catalog_manager.cc:4985] TS 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889): tablet bb4aebbafbe6415e90eaf6f6feb6b760 (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:28.283138 32550 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:28.283221 32550 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:28.283411 31795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:28.283493 31795 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:28.283535 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:28.284124 31795 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:28.284307 31795 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 5 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:28.284456 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:28.284456 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:28.284564 31719 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Stepping down as leader of term 4
I20251027 03:09:28.284574 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:28.284590 31719 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:28.284663 31719 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:28.284750 31719 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:28.285192 31639 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 5.
I20251027 03:09:28.285301 31719 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 5.
I20251027 03:09:28.285395 31760 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:28.285485 32500 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Leader election won for term 5
I20251027 03:09:28.285538 32500 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:28.285594 32500 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:28.286119 31539 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 4 to 5, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 5 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:28.317237 32128 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:28.317482 32550 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.318601 31795 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:28.318756 32550 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.487191 32051 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEARNER]: Advancing to term 3
I20251027 03:09:28.487895 32051 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:28.488123 32555 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:28.488914 32555 raft_consensus.cc:1064] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
W20251027 03:09:28.488967 32555 raft_consensus.cc:1067] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Unable to promote non-voter 93e976cbba5e4213a43042ffe2da42d6: Service unavailable: leader transfer in progress
I20251027 03:09:28.491955 31639 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20251027 03:09:28.492172 32545 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:28.493043 31795 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20251027 03:09:28.493192 32545 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:28.495503 31719 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:28.495667 32555 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:28.496387 32552 raft_consensus.cc:993] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:28.496443 32552 raft_consensus.cc:1081] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:28.496713 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251027 03:09:28.496943 31719 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:28.497006 31719 raft_consensus.cc:493] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:28.497035 31719 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:28.497195 32552 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:28.497813 31719 raft_consensus.cc:515] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:28.497932 31719 leader_election.cc:290] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 4 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:28.497038 32502 raft_consensus.cc:1064] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:28.498049 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:28.498133 31639 raft_consensus.cc:3055] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:28.498167 31639 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:28.498229 31639 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:28.498296 31639 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:28.498735 31639 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 4.
W20251027 03:09:28.498059 32502 raft_consensus.cc:1067] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Unable to promote non-voter 93e976cbba5e4213a43042ffe2da42d6: Service unavailable: leader transfer in progress
I20251027 03:09:28.498023 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:28.498885 31685 leader_election.cc:304] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:28.498893 31795 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:28.498970 32568 raft_consensus.cc:2804] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:28.499013 32568 raft_consensus.cc:697] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:28.499063 32568 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:28.499477 31795 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 4.
I20251027 03:09:28.499437 31542 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 3 to 4, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 4 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:28.661697 31719 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20251027 03:09:28.661901 32500 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.662848 31639 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20251027 03:09:28.663029 32561 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.875597 31639 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251027 03:09:28.875873 32569 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.876833 32051 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 LEARNER]: Advancing to term 4
I20251027 03:09:28.876853 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251027 03:09:28.877456 32550 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.877687 32051 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEARNER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251027 03:09:28.877964 32550 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:28.878458 32569 raft_consensus.cc:1064] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:28.878611 32569 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 31, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:28.878991 32051 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEARNER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:28.879014 31639 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:28.879040 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20251027 03:09:28.879207 32589 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.879274 32568 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.879328 32550 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:28.879849 32550 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.880062 32051 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.880560 31639 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.882478 31539 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: config changed from index 30 to 33, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.882859 31795 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.883349 31719 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:28.883667 32051 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20251027 03:09:28.883855 32569 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:28.884032 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20251027 03:09:28.884222 32569 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:28.884397 32589 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.884516 32051 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.884930 31795 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.885586 31542 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: config changed from index 33 to 34, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New cstate: current_term: 4 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:28.885735 31529 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 11ce7fc7b4334ffe99df2944dca2af4a with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:28.886252 31635 tablet_service.cc:1552] Processing DeleteTablet for tablet 11ce7fc7b4334ffe99df2944dca2af4a with delete_type TABLET_DATA_TOMBSTONED (TS 78e34a95a4a44fde91fa13a71fa3da26 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:28.886425 32590 tablet_replica.cc:333] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:28.886493 32590 raft_consensus.cc:2243] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:28.886543 32590 raft_consensus.cc:2272] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:28.886762 32590 ts_tablet_manager.cc:1916] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:28.887725 32590 ts_tablet_manager.cc:1929] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20251027 03:09:28.887775 32590 log.cc:1199] T 11ce7fc7b4334ffe99df2944dca2af4a P 78e34a95a4a44fde91fa13a71fa3da26: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/11ce7fc7b4334ffe99df2944dca2af4a
I20251027 03:09:28.888041 31527 catalog_manager.cc:4985] TS 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889): tablet 11ce7fc7b4334ffe99df2944dca2af4a (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:29.213112 31639 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:29.213616 31719 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20251027 03:09:29.213618 32128 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
W20251027 03:09:29.213683 31611 consensus_peers.cc:597] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30ef3ff09aa247fab2eae47a3f1718fd. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:29.213809 32594 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:29.213968 32594 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:29.214502 32593 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:29.214614 31719 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:29.214874 32128 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:29.215631 31539 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index 33 to 34, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:29.216075 31719 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:29.216434 32051 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20251027 03:09:29.216631 32568 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:29.216722 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
W20251027 03:09:29.216777 31686 consensus_peers.cc:597] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11ce7fc7b4334ffe99df2944dca2af4a. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:29.216871 32568 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:29.217265 32569 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 34 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.217444 31795 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.218124 32051 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.218323 31538 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index 34 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New cstate: current_term: 4 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:29.221509 32128 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:29.222012 31719 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20251027 03:09:29.222186 31795 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20251027 03:09:29.222206 32601 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:29.222420 32601 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:29.222828 32602 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 33 to 35, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:29.222919 31719 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 33 to 35, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:29.223703 31542 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 reported cstate change: config changed from index 33 to 35, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New cstate: current_term: 4 leader_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:29.223871 32202 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:29.224228 31795 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 33 to 35, NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } } }
I20251027 03:09:29.224514 31639 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20251027 03:09:29.224583 31792 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20251027 03:09:29.224726 32591 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:29.224835 32591 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20251027 03:09:29.225584 32094 consensus_peers.cc:597] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 -> Peer 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331): Couldn't send request to peer 93e976cbba5e4213a43042ffe2da42d6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e2464cf0e9243b7845c98d8cde8528b. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:29.225617 31639 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 33 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.225509 32545 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 LEADER]: Committing config change with OpId 5.35: config changed from index 33 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.226577 31542 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 33 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New cstate: current_term: 5 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.226656 31795 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Committing config change with OpId 5.35: config changed from index 33 to 35, NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } } }
I20251027 03:09:29.227013 31719 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:29.227406 32051 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251027 03:09:29.227599 32568 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:29.227902 31639 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
W20251027 03:09:29.228049 31687 consensus_peers.cc:597] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f -> Peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021): Couldn't send request to peer d421b303d20a48cfbbd34da50b8d0d6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c176d9f872b84ac986522990949395eb. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:29.228119 32568 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251027 03:09:29.228618 32550 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:29.228761 32051 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
I20251027 03:09:29.228910 31639 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } } }
W20251027 03:09:29.229377 32167 consensus_peers.cc:597] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e1ec5a8f7224329be0fe0e5453d71fc. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:29.229651 31542 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: config changed from index 29 to 30, NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) added. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:29.604238 32612 ts_tablet_manager.cc:933] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: Initiating tablet copy from peer 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:29.604437 32612 tablet_copy_client.cc:323] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:41091
I20251027 03:09:29.604651 31732 tablet_copy_service.cc:140] P 50aa97f11c164e0b9e3520267cf6c85f: Received BeginTabletCopySession request for tablet c176d9f872b84ac986522990949395eb from peer d421b303d20a48cfbbd34da50b8d0d6d ({username='slave'} at 127.0.0.1:57754)
I20251027 03:09:29.604709 31732 tablet_copy_service.cc:161] P 50aa97f11c164e0b9e3520267cf6c85f: Beginning new tablet copy session on tablet c176d9f872b84ac986522990949395eb from peer d421b303d20a48cfbbd34da50b8d0d6d at {username='slave'} at 127.0.0.1:57754: session id = d421b303d20a48cfbbd34da50b8d0d6d-c176d9f872b84ac986522990949395eb
I20251027 03:09:29.605333 31732 tablet_copy_source_session.cc:215] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:29.605705 32612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c176d9f872b84ac986522990949395eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:29.607167 32612 tablet_copy_client.cc:806] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:29.607292 32612 tablet_copy_client.cc:670] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:29.607851 32613 tablet_copy_client.cc:1025] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:29.660944 32614 ts_tablet_manager.cc:933] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: Initiating tablet copy from peer 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:29.661126 32614 tablet_copy_client.cc:323] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:41091
I20251027 03:09:29.661307 31732 tablet_copy_service.cc:140] P 50aa97f11c164e0b9e3520267cf6c85f: Received BeginTabletCopySession request for tablet 11ce7fc7b4334ffe99df2944dca2af4a from peer bcd3eff6ab6f42e9b6b475710ba14c80 ({username='slave'} at 127.0.0.1:57758)
I20251027 03:09:29.661356 31732 tablet_copy_service.cc:161] P 50aa97f11c164e0b9e3520267cf6c85f: Beginning new tablet copy session on tablet 11ce7fc7b4334ffe99df2944dca2af4a from peer bcd3eff6ab6f42e9b6b475710ba14c80 at {username='slave'} at 127.0.0.1:57758: session id = bcd3eff6ab6f42e9b6b475710ba14c80-11ce7fc7b4334ffe99df2944dca2af4a
I20251027 03:09:29.661937 31732 tablet_copy_source_session.cc:215] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:29.662242 32614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ce7fc7b4334ffe99df2944dca2af4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:29.663801 32614 tablet_copy_client.cc:806] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:29.663919 32614 tablet_copy_client.cc:670] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:29.664448 32615 tablet_copy_client.cc:1025] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:29.715117 32616 ts_tablet_manager.cc:933] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: Initiating tablet copy from peer d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:29.715283 32616 tablet_copy_client.cc:323] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.70:42021
I20251027 03:09:29.718247 32213 tablet_copy_service.cc:140] P d421b303d20a48cfbbd34da50b8d0d6d: Received BeginTabletCopySession request for tablet 5e1ec5a8f7224329be0fe0e5453d71fc from peer bcd3eff6ab6f42e9b6b475710ba14c80 ({username='slave'} at 127.0.0.1:47852)
I20251027 03:09:29.718313 32213 tablet_copy_service.cc:161] P d421b303d20a48cfbbd34da50b8d0d6d: Beginning new tablet copy session on tablet 5e1ec5a8f7224329be0fe0e5453d71fc from peer bcd3eff6ab6f42e9b6b475710ba14c80 at {username='slave'} at 127.0.0.1:47852: session id = bcd3eff6ab6f42e9b6b475710ba14c80-5e1ec5a8f7224329be0fe0e5453d71fc
I20251027 03:09:29.718869 32213 tablet_copy_source_session.cc:215] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:29.719162 32616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e1ec5a8f7224329be0fe0e5453d71fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:29.720589 32616 tablet_copy_client.cc:806] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:29.720721 32616 tablet_copy_client.cc:670] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:29.721232 32617 tablet_copy_client.cc:1025] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:29.725059 32619 ts_tablet_manager.cc:933] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: Initiating tablet copy from peer 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889)
I20251027 03:09:29.725191 32619 tablet_copy_client.cc:323] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:32889
I20251027 03:09:29.725351 31656 tablet_copy_service.cc:140] P 78e34a95a4a44fde91fa13a71fa3da26: Received BeginTabletCopySession request for tablet 30ef3ff09aa247fab2eae47a3f1718fd from peer d421b303d20a48cfbbd34da50b8d0d6d ({username='slave'} at 127.0.0.1:43414)
I20251027 03:09:29.725399 31656 tablet_copy_service.cc:161] P 78e34a95a4a44fde91fa13a71fa3da26: Beginning new tablet copy session on tablet 30ef3ff09aa247fab2eae47a3f1718fd from peer d421b303d20a48cfbbd34da50b8d0d6d at {username='slave'} at 127.0.0.1:43414: session id = d421b303d20a48cfbbd34da50b8d0d6d-30ef3ff09aa247fab2eae47a3f1718fd
I20251027 03:09:29.725852 31656 tablet_copy_source_session.cc:215] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:29.726148 32619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30ef3ff09aa247fab2eae47a3f1718fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:29.727389 32619 tablet_copy_client.cc:806] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:29.727551 32619 tablet_copy_client.cc:670] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:29.728006 32620 tablet_copy_client.cc:1025] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:29.820111 32621 ts_tablet_manager.cc:933] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: Initiating tablet copy from peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233)
I20251027 03:09:29.820309 32621 tablet_copy_client.cc:323] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.69:42233
I20251027 03:09:29.822937 32138 tablet_copy_service.cc:140] P bcd3eff6ab6f42e9b6b475710ba14c80: Received BeginTabletCopySession request for tablet 0e2464cf0e9243b7845c98d8cde8528b from peer 93e976cbba5e4213a43042ffe2da42d6 ({username='slave'} at 127.0.0.1:58178)
I20251027 03:09:29.822994 32138 tablet_copy_service.cc:161] P bcd3eff6ab6f42e9b6b475710ba14c80: Beginning new tablet copy session on tablet 0e2464cf0e9243b7845c98d8cde8528b from peer 93e976cbba5e4213a43042ffe2da42d6 at {username='slave'} at 127.0.0.1:58178: session id = 93e976cbba5e4213a43042ffe2da42d6-0e2464cf0e9243b7845c98d8cde8528b
I20251027 03:09:29.823602 32138 tablet_copy_source_session.cc:215] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:29.823925 32621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2464cf0e9243b7845c98d8cde8528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:29.825098 32621 tablet_copy_client.cc:806] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:29.825212 32621 tablet_copy_client.cc:670] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:29.825692 32622 tablet_copy_client.cc:1025] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Injecting latency into file download: 2000
I20251027 03:09:29.878453 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:29.878865 31719 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "93e976cbba5e4213a43042ffe2da42d6"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:29.878966 31719 raft_consensus.cc:606] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Received request to transfer leadership to TS 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:29.879257 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:29.879333 31795 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:29.879657 31719 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:29.879721 31719 raft_consensus.cc:606] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Received request to transfer leadership to TS 10153e76038242fdaec0d401cdbb0411
I20251027 03:09:29.879841 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251027 03:09:29.879910 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:30.107525 32625 raft_consensus.cc:993] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 93e976cbba5e4213a43042ffe2da42d6 to start an election
I20251027 03:09:30.107673 32625 raft_consensus.cc:1081] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Signalling peer 93e976cbba5e4213a43042ffe2da42d6 to start an election
I20251027 03:09:30.107846 32051 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a"
dest_uuid: "93e976cbba5e4213a43042ffe2da42d6"
from {username='slave'} at 127.0.0.1:56724
I20251027 03:09:30.107914 32051 raft_consensus.cc:493] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:30.107940 32051 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:30.108726 32051 raft_consensus.cc:515] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:30.108901 32051 leader_election.cc:290] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [CANDIDATE]: Term 5 election: Requested vote from peers 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045), 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091)
I20251027 03:09:30.108996 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "93e976cbba5e4213a43042ffe2da42d6" candidate_term: 5 candidate_status { last_received { term: 4 index: 35 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:30.109004 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11ce7fc7b4334ffe99df2944dca2af4a" candidate_uuid: "93e976cbba5e4213a43042ffe2da42d6" candidate_term: 5 candidate_status { last_received { term: 4 index: 35 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:30.109089 31795 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:30.109109 31719 raft_consensus.cc:3055] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Stepping down as leader of term 4
I20251027 03:09:30.109148 31719 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:30.109208 31719 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 35, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:30.109288 31719 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:30.109786 31795 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e976cbba5e4213a43042ffe2da42d6 in term 5.
I20251027 03:09:30.109907 31719 raft_consensus.cc:2468] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e976cbba5e4213a43042ffe2da42d6 in term 5.
I20251027 03:09:30.109964 32015 leader_election.cc:304] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 93e976cbba5e4213a43042ffe2da42d6; no voters:
I20251027 03:09:30.110157 32626 raft_consensus.cc:2804] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 FOLLOWER]: Leader election won for term 5
I20251027 03:09:30.110298 32626 raft_consensus.cc:697] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 LEADER]: Becoming Leader. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Running, Role: LEADER
I20251027 03:09:30.110378 32626 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 35, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:30.110910 31538 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: term changed from 4 to 5, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68). New cstate: current_term: 5 leader_uuid: "93e976cbba5e4213a43042ffe2da42d6" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:30.193862 32563 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:30.193943 32563 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:30.194077 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43380
I20251027 03:09:30.194157 31639 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:30.194190 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Advancing to term 6
I20251027 03:09:30.195019 31639 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:30.195196 31639 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 6 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:30.195328 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:30.195328 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:30.195433 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Advancing to term 6
I20251027 03:09:30.195438 31795 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Stepping down as leader of term 5
I20251027 03:09:30.195484 31795 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:30.195544 31795 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:30.195622 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Advancing to term 6
I20251027 03:09:30.196241 31717 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 6.
I20251027 03:09:30.196266 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 6.
I20251027 03:09:30.196436 31611 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:30.196528 32634 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 FOLLOWER]: Leader election won for term 6
I20251027 03:09:30.196585 32634 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:30.196640 32634 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:30.197129 31538 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 5 to 6, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 6 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:30.319243 32568 raft_consensus.cc:993] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:30.319339 32568 raft_consensus.cc:1081] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Signalling peer 10153e76038242fdaec0d401cdbb0411 to start an election
I20251027 03:09:30.319491 31795 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
from {username='slave'} at 127.0.0.1:52982
I20251027 03:09:30.319571 31795 raft_consensus.cc:493] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:30.319594 31795 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:30.320464 31795 raft_consensus.cc:515] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:30.320679 31795 leader_election.cc:290] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 5 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233)
I20251027 03:09:30.320794 32128 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80"
I20251027 03:09:30.320863 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726dd98dfd6c47a5b95b723de71cb5c5" candidate_uuid: "10153e76038242fdaec0d401cdbb0411" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:30.320871 32128 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:30.320942 31719 raft_consensus.cc:3055] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Stepping down as leader of term 4
I20251027 03:09:30.320968 31719 raft_consensus.cc:740] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:30.321039 31719 consensus_queue.cc:260] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:30.321113 31719 raft_consensus.cc:3060] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Advancing to term 5
I20251027 03:09:30.321617 32128 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 5.
I20251027 03:09:30.321741 31761 leader_election.cc:304] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, bcd3eff6ab6f42e9b6b475710ba14c80; no voters:
I20251027 03:09:30.321776 31719 raft_consensus.cc:2468] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10153e76038242fdaec0d401cdbb0411 in term 5.
I20251027 03:09:30.321873 32635 raft_consensus.cc:2804] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Leader election won for term 5
I20251027 03:09:30.321918 32635 raft_consensus.cc:697] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Becoming Leader. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:30.321972 32635 consensus_queue.cc:237] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:30.322604 31538 catalog_manager.cc:5649] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 reported cstate change: term changed from 4 to 5, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 10153e76038242fdaec0d401cdbb0411 (127.27.151.67). New cstate: current_term: 5 leader_uuid: "10153e76038242fdaec0d401cdbb0411" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:30.509827 31719 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20251027 03:09:30.510088 32626 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:30.511262 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20251027 03:09:30.511477 32643 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
W20251027 03:09:30.511827 32016 consensus_peers.cc:597] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 -> Peer bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69:42233): Couldn't send request to peer bcd3eff6ab6f42e9b6b475710ba14c80. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:30.597290 31719 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 6 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20251027 03:09:30.597492 32634 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:30.598654 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 6 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20251027 03:09:30.598976 32634 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251027 03:09:30.766233 31717 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20251027 03:09:30.766420 32563 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:30.767200 32128 raft_consensus.cc:1275] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Refusing update from remote peer 10153e76038242fdaec0d401cdbb0411: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20251027 03:09:30.767375 32563 consensus_queue.cc:1048] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:31.608074 32612 tablet_copy_client.cc:538] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:31.609037 32612 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap starting.
I20251027 03:09:31.612442 32612 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:31.612636 32612 tablet_bootstrap.cc:492] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap complete.
I20251027 03:09:31.612758 32612 ts_tablet_manager.cc:1403] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20251027 03:09:31.612908 32612 raft_consensus.cc:359] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:31.612994 32612 raft_consensus.cc:740] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Initialized, Role: LEARNER
I20251027 03:09:31.613035 32612 consensus_queue.cc:260] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:31.613135 32612 ts_tablet_manager.cc:1434] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:09:31.613317 31732 tablet_copy_service.cc:342] P 50aa97f11c164e0b9e3520267cf6c85f: Request end of tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-c176d9f872b84ac986522990949395eb received from {username='slave'} at 127.0.0.1:57754
I20251027 03:09:31.613379 31732 tablet_copy_service.cc:434] P 50aa97f11c164e0b9e3520267cf6c85f: ending tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-c176d9f872b84ac986522990949395eb on tablet c176d9f872b84ac986522990949395eb with peer d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:31.664646 32614 tablet_copy_client.cc:538] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:31.665547 32614 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap starting.
I20251027 03:09:31.670117 32614 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:31.670305 32614 tablet_bootstrap.cc:492] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap complete.
I20251027 03:09:31.670436 32614 ts_tablet_manager.cc:1403] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251027 03:09:31.670588 32614 raft_consensus.cc:359] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:31.670666 32614 raft_consensus.cc:740] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Initialized, Role: LEARNER
I20251027 03:09:31.670732 32614 consensus_queue.cc:260] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 4.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:31.670836 32614 ts_tablet_manager.cc:1434] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s
I20251027 03:09:31.670992 31732 tablet_copy_service.cc:342] P 50aa97f11c164e0b9e3520267cf6c85f: Request end of tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-11ce7fc7b4334ffe99df2944dca2af4a received from {username='slave'} at 127.0.0.1:57758
I20251027 03:09:31.671033 31732 tablet_copy_service.cc:434] P 50aa97f11c164e0b9e3520267cf6c85f: ending tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-11ce7fc7b4334ffe99df2944dca2af4a on tablet 11ce7fc7b4334ffe99df2944dca2af4a with peer bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:31.679900 32203 raft_consensus.cc:1217] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[]
I20251027 03:09:31.721408 32616 tablet_copy_client.cc:538] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:31.722337 32616 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap starting.
I20251027 03:09:31.725586 32616 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=109 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:31.725770 32616 tablet_bootstrap.cc:492] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: Bootstrap complete.
I20251027 03:09:31.725858 32616 ts_tablet_manager.cc:1403] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 03:09:31.726001 32616 raft_consensus.cc:359] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:31.726070 32616 raft_consensus.cc:740] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: bcd3eff6ab6f42e9b6b475710ba14c80, State: Initialized, Role: LEARNER
I20251027 03:09:31.726089 32616 consensus_queue.cc:260] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: NON_VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: true } }
I20251027 03:09:31.726226 32616 ts_tablet_manager.cc:1434] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s
I20251027 03:09:31.726464 32213 tablet_copy_service.cc:342] P d421b303d20a48cfbbd34da50b8d0d6d: Request end of tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-5e1ec5a8f7224329be0fe0e5453d71fc received from {username='slave'} at 127.0.0.1:47852
I20251027 03:09:31.726507 32213 tablet_copy_service.cc:434] P d421b303d20a48cfbbd34da50b8d0d6d: ending tablet copy session bcd3eff6ab6f42e9b6b475710ba14c80-5e1ec5a8f7224329be0fe0e5453d71fc on tablet 5e1ec5a8f7224329be0fe0e5453d71fc with peer bcd3eff6ab6f42e9b6b475710ba14c80
I20251027 03:09:31.728219 32619 tablet_copy_client.cc:538] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:31.728925 32619 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap starting.
I20251027 03:09:31.733357 32619 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=133 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:31.733539 32619 tablet_bootstrap.cc:492] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: Bootstrap complete.
I20251027 03:09:31.733652 32619 ts_tablet_manager.cc:1403] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251027 03:09:31.733806 32619 raft_consensus.cc:359] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:31.733886 32619 raft_consensus.cc:740] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: d421b303d20a48cfbbd34da50b8d0d6d, State: Initialized, Role: LEARNER
I20251027 03:09:31.733927 32619 consensus_queue.cc:260] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: NON_VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: true } }
I20251027 03:09:31.734040 32619 ts_tablet_manager.cc:1434] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:31.734179 31656 tablet_copy_service.cc:342] P 78e34a95a4a44fde91fa13a71fa3da26: Request end of tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-30ef3ff09aa247fab2eae47a3f1718fd received from {username='slave'} at 127.0.0.1:43414
I20251027 03:09:31.734233 31656 tablet_copy_service.cc:434] P 78e34a95a4a44fde91fa13a71fa3da26: ending tablet copy session d421b303d20a48cfbbd34da50b8d0d6d-30ef3ff09aa247fab2eae47a3f1718fd on tablet 30ef3ff09aa247fab2eae47a3f1718fd with peer d421b303d20a48cfbbd34da50b8d0d6d
I20251027 03:09:31.825897 32621 tablet_copy_client.cc:538] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:31.826946 32621 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap starting.
I20251027 03:09:31.831897 32621 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=119 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:31.832098 32621 tablet_bootstrap.cc:492] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: Bootstrap complete.
I20251027 03:09:31.832227 32621 ts_tablet_manager.cc:1403] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.001s
I20251027 03:09:31.832373 32621 raft_consensus.cc:359] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:31.832450 32621 raft_consensus.cc:740] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Initialized, Role: LEARNER
I20251027 03:09:31.832517 32621 consensus_queue.cc:260] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 4.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: true } }
I20251027 03:09:31.832731 32621 ts_tablet_manager.cc:1434] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:31.832939 32138 tablet_copy_service.cc:342] P bcd3eff6ab6f42e9b6b475710ba14c80: Request end of tablet copy session 93e976cbba5e4213a43042ffe2da42d6-0e2464cf0e9243b7845c98d8cde8528b received from {username='slave'} at 127.0.0.1:58178
I20251027 03:09:31.833003 32138 tablet_copy_service.cc:434] P bcd3eff6ab6f42e9b6b475710ba14c80: ending tablet copy session 93e976cbba5e4213a43042ffe2da42d6-0e2464cf0e9243b7845c98d8cde8528b on tablet 0e2464cf0e9243b7845c98d8cde8528b with peer 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:31.880029 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:31.880399 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:31.880496 31639 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:31.880816 31795 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a"
dest_uuid: "10153e76038242fdaec0d401cdbb0411"
mode: GRACEFUL
new_leader_uuid: "93e976cbba5e4213a43042ffe2da42d6"
from {username='slave'} at 127.0.0.1:53010
I20251027 03:09:31.880887 31795 raft_consensus.cc:606] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Received request to transfer leadership to TS 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:31.881021 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251027 03:09:31.881086 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:31.910857 32128 raft_consensus.cc:1217] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.34->[5.35-5.35] Dedup: 5.35->[]
I20251027 03:09:32.065171 32128 raft_consensus.cc:3060] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEARNER]: Advancing to term 5
I20251027 03:09:32.133133 32661 raft_consensus.cc:993] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: : Instructing follower 93e976cbba5e4213a43042ffe2da42d6 to start an election
I20251027 03:09:32.133221 32661 raft_consensus.cc:1081] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Signalling peer 93e976cbba5e4213a43042ffe2da42d6 to start an election
I20251027 03:09:32.133412 32052 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a"
dest_uuid: "93e976cbba5e4213a43042ffe2da42d6"
from {username='slave'} at 127.0.0.1:35998
I20251027 03:09:32.133497 32052 raft_consensus.cc:493] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:32.133526 32052 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:32.134234 32052 raft_consensus.cc:515] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:32.134366 32052 leader_election.cc:290] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [CANDIDATE]: Term 4 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:32.134503 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "93e976cbba5e4213a43042ffe2da42d6" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:32.134513 31719 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7a4ca641cf349f29eeb0292a996449a" candidate_uuid: "93e976cbba5e4213a43042ffe2da42d6" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:32.134573 31795 raft_consensus.cc:3055] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:32.134588 31719 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:32.134595 31795 raft_consensus.cc:740] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 10153e76038242fdaec0d401cdbb0411, State: Running, Role: LEADER
I20251027 03:09:32.134701 31795 consensus_queue.cc:260] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:32.134773 31795 raft_consensus.cc:3060] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:32.135301 31719 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e976cbba5e4213a43042ffe2da42d6 in term 4.
I20251027 03:09:32.135411 31795 raft_consensus.cc:2468] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e976cbba5e4213a43042ffe2da42d6 in term 4.
I20251027 03:09:32.135623 32015 leader_election.cc:304] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 93e976cbba5e4213a43042ffe2da42d6; no voters:
I20251027 03:09:32.135730 32643 raft_consensus.cc:2804] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:32.135861 32643 raft_consensus.cc:697] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEADER]: Becoming Leader. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Running, Role: LEADER
I20251027 03:09:32.135929 32643 consensus_queue.cc:237] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:32.136422 31538 catalog_manager.cc:5649] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: term changed from 3 to 4, leader changed from 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) to 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68). New cstate: current_term: 4 leader_uuid: "93e976cbba5e4213a43042ffe2da42d6" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:32.148226 32203 raft_consensus.cc:1217] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 LEARNER]: Deduplicated request from leader. Original: 3.33->[3.34-3.34] Dedup: 3.34->[]
I20251027 03:09:32.190871 32651 raft_consensus.cc:1064] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: attempting to promote NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d to VOTER
I20251027 03:09:32.191063 32651 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:32.191541 32203 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 LEARNER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:32.191597 31640 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:32.191725 32676 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:32.191778 32052 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20251027 03:09:32.191823 32651 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:32.191915 32651 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251027 03:09:32.192560 32676 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.192660 31640 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.192670 32052 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.193631 31538 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 30 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.193836 32203 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.195022 31717 consensus_queue.cc:237] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:32.195411 32051 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251027 03:09:32.195452 32203 raft_consensus.cc:1275] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20251027 03:09:32.195585 32676 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:32.195717 32651 consensus_queue.cc:1048] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251027 03:09:32.196318 32678 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.196489 32202 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.197214 31529 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c176d9f872b84ac986522990949395eb with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:32.197290 31538 catalog_manager.cc:5649] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: config changed from index 31 to 32, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New cstate: current_term: 1 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.197630 32051 raft_consensus.cc:2955] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.198580 31635 tablet_service.cc:1552] Processing DeleteTablet for tablet c176d9f872b84ac986522990949395eb with delete_type TABLET_DATA_TOMBSTONED (TS 78e34a95a4a44fde91fa13a71fa3da26 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:43356
I20251027 03:09:32.198772 32682 tablet_replica.cc:333] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:32.198859 32682 raft_consensus.cc:2243] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:32.198975 32682 raft_consensus.cc:2272] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:32.199210 32682 ts_tablet_manager.cc:1916] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:32.200569 32682 ts_tablet_manager.cc:1929] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.31
I20251027 03:09:32.200644 32682 log.cc:1199] T c176d9f872b84ac986522990949395eb P 78e34a95a4a44fde91fa13a71fa3da26: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/c176d9f872b84ac986522990949395eb
I20251027 03:09:32.200953 31527 catalog_manager.cc:4985] TS 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889): tablet c176d9f872b84ac986522990949395eb (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:32.289036 32051 raft_consensus.cc:1217] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.34->[4.35-4.35] Dedup: 4.35->[]
I20251027 03:09:32.358232 32650 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:32.358322 32650 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:32.358459 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:32.358549 31717 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:32.358578 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 6 FOLLOWER]: Advancing to term 7
I20251027 03:09:32.359401 31717 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:32.359575 31717 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 7 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:32.359820 31640 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 7 candidate_status { last_received { term: 6 index: 33 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:32.359887 31640 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 LEADER]: Stepping down as leader of term 6
I20251027 03:09:32.359920 31640 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:32.359978 31640 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:32.360037 31640 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 6 FOLLOWER]: Advancing to term 7
I20251027 03:09:32.360452 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 7 candidate_status { last_received { term: 6 index: 33 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:32.360524 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 6 FOLLOWER]: Advancing to term 7
I20251027 03:09:32.360705 31640 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 7.
I20251027 03:09:32.360884 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50aa97f11c164e0b9e3520267cf6c85f, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:32.361220 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 7.
I20251027 03:09:32.361304 32651 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 FOLLOWER]: Leader election won for term 7
I20251027 03:09:32.361348 32651 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:32.361406 32651 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 33, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:32.362092 31542 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 6 to 7, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 7 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:32.500761 32657 raft_consensus.cc:1064] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
I20251027 03:09:32.500970 32657 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 33, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:32.501427 32127 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 LEARNER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20251027 03:09:32.501497 31640 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20251027 03:09:32.501497 31795 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20251027 03:09:32.501595 32688 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.501745 32687 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.501824 32657 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.502678 32657 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 LEADER]: Committing config change with OpId 5.36: config changed from index 35 to 36, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.502774 31795 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.503600 31640 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.503664 31542 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: config changed from index 35 to 36, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:32.502774 32127 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 35 to 36, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.504498 32202 consensus_queue.cc:237] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 5.36, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:32.505013 31640 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20251027 03:09:32.505167 32688 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.505728 32128 raft_consensus.cc:1275] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Refusing update from remote peer d421b303d20a48cfbbd34da50b8d0d6d: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20251027 03:09:32.505825 32685 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 LEADER]: Committing config change with OpId 5.37: config changed from index 36 to 37, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.506160 31639 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.506280 32657 consensus_queue.cc:1048] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.506531 32128 raft_consensus.cc:2955] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.506681 31590 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 5e1ec5a8f7224329be0fe0e5453d71fc with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:32.507449 31542 catalog_manager.cc:5649] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 36 to 37, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New cstate: current_term: 5 leader_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.508083 31779 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e1ec5a8f7224329be0fe0e5453d71fc with delete_type TABLET_DATA_TOMBSTONED (TS 10153e76038242fdaec0d401cdbb0411 not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:52962
I20251027 03:09:32.508311 32692 tablet_replica.cc:333] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:32.508421 32692 raft_consensus.cc:2243] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:32.508491 32692 raft_consensus.cc:2272] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:32.508739 32692 ts_tablet_manager.cc:1916] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:32.509820 32692 ts_tablet_manager.cc:1929] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.36
I20251027 03:09:32.509878 32692 log.cc:1199] T 5e1ec5a8f7224329be0fe0e5453d71fc P 10153e76038242fdaec0d401cdbb0411: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/5e1ec5a8f7224329be0fe0e5453d71fc
I20251027 03:09:32.510136 31527 catalog_manager.cc:4985] TS 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045): tablet 5e1ec5a8f7224329be0fe0e5453d71fc (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:32.584349 31717 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20251027 03:09:32.584546 32643 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:32.585656 31795 raft_consensus.cc:1275] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20251027 03:09:32.585858 32670 consensus_queue.cc:1048] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:32.608963 32643 raft_consensus.cc:1064] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: attempting to promote NON_VOTER bcd3eff6ab6f42e9b6b475710ba14c80 to VOTER
I20251027 03:09:32.609143 32643 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 5.36, Last appended by leader: 35, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:32.609546 32128 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 LEARNER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20251027 03:09:32.609624 31795 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20251027 03:09:32.609625 31717 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20251027 03:09:32.609755 32698 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.609952 32700 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.610148 32670 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.610808 32698 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 LEADER]: Committing config change with OpId 5.37: config changed from index 35 to 37, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.610951 31795 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 35 to 37, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.611899 31717 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 35 to 37, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.612286 31539 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 35 to 37, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "93e976cbba5e4213a43042ffe2da42d6" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.612344 32128 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 35 to 37, bcd3eff6ab6f42e9b6b475710ba14c80 (127.27.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } attrs { replace: true } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.613662 32051 consensus_queue.cc:237] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 5.37, Last appended by leader: 35, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } }
I20251027 03:09:32.613976 32127 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20251027 03:09:32.613992 31717 raft_consensus.cc:1275] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20251027 03:09:32.614140 32700 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251027 03:09:32.614210 32643 consensus_queue.cc:1048] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251027 03:09:32.614802 32643 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 LEADER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.614895 31717 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.615656 32127 raft_consensus.cc:2955] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } }
I20251027 03:09:32.615700 31542 catalog_manager.cc:5649] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: config changed from index 37 to 38, VOTER 10153e76038242fdaec0d401cdbb0411 (127.27.151.67) evicted. New cstate: current_term: 5 leader_uuid: "93e976cbba5e4213a43042ffe2da42d6" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:32.615692 31591 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 11ce7fc7b4334ffe99df2944dca2af4a with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:32.616407 31779 tablet_service.cc:1552] Processing DeleteTablet for tablet 11ce7fc7b4334ffe99df2944dca2af4a with delete_type TABLET_DATA_TOMBSTONED (TS 10153e76038242fdaec0d401cdbb0411 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:52962
I20251027 03:09:32.616511 32692 tablet_replica.cc:333] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:32.616576 32692 raft_consensus.cc:2243] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:32.616647 32692 raft_consensus.cc:2272] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:32.617084 32692 ts_tablet_manager.cc:1916] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:32.618229 32692 ts_tablet_manager.cc:1929] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.37
I20251027 03:09:32.618287 32692 log.cc:1199] T 11ce7fc7b4334ffe99df2944dca2af4a P 10153e76038242fdaec0d401cdbb0411: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/11ce7fc7b4334ffe99df2944dca2af4a
I20251027 03:09:32.618568 31527 catalog_manager.cc:4985] TS 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045): tablet 11ce7fc7b4334ffe99df2944dca2af4a (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:32.733601 32650 raft_consensus.cc:1064] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: attempting to promote NON_VOTER d421b303d20a48cfbbd34da50b8d0d6d to VOTER
I20251027 03:09:32.733822 32650 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:32.734313 32128 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20251027 03:09:32.734339 32202 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 LEARNER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20251027 03:09:32.734398 31717 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20251027 03:09:32.734539 32683 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:32.734620 32650 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:32.734740 32705 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:32.735700 32683 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Committing config change with OpId 3.35: config changed from index 34 to 35, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.735728 32202 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.736680 31542 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: config changed from index 34 to 35, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:32.736701 31717 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.736980 32128 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.737468 31639 consensus_queue.cc:237] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 3.35, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:32.737860 32202 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20251027 03:09:32.738032 32704 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.738088 32127 raft_consensus.cc:1275] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20251027 03:09:32.738269 32704 consensus_queue.cc:1048] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.738595 32704 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.738713 32202 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.738771 32128 raft_consensus.cc:2955] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.739475 31527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 30ef3ff09aa247fab2eae47a3f1718fd with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:32.739598 31538 catalog_manager.cc:5649] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d reported cstate change: config changed from index 35 to 36, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } } }
I20251027 03:09:32.740278 31704 tablet_service.cc:1552] Processing DeleteTablet for tablet 30ef3ff09aa247fab2eae47a3f1718fd with delete_type TABLET_DATA_TOMBSTONED (TS 50aa97f11c164e0b9e3520267cf6c85f not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:32.740475 32709 tablet_replica.cc:333] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:32.740563 32709 raft_consensus.cc:2243] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:32.740660 32709 raft_consensus.cc:2272] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:32.740916 32709 ts_tablet_manager.cc:1916] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:32.742300 32709 ts_tablet_manager.cc:1929] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.35
I20251027 03:09:32.742357 32709 log.cc:1199] T 30ef3ff09aa247fab2eae47a3f1718fd P 50aa97f11c164e0b9e3520267cf6c85f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/30ef3ff09aa247fab2eae47a3f1718fd
I20251027 03:09:32.742681 31529 catalog_manager.cc:4985] TS 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091): tablet 30ef3ff09aa247fab2eae47a3f1718fd (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:32.760286 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 7 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 7 index: 34. (index mismatch)
I20251027 03:09:32.760490 32651 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:32.761241 31639 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 7 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 7 index: 34. (index mismatch)
I20251027 03:09:32.761446 32678 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251027 03:09:32.815399 32714 raft_consensus.cc:1064] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: attempting to promote NON_VOTER 93e976cbba5e4213a43042ffe2da42d6 to VOTER
I20251027 03:09:32.815667 32714 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 33, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:32.816116 32051 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEARNER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20251027 03:09:32.816216 31717 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20251027 03:09:32.816293 32713 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.816394 32714 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.816308 31795 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20251027 03:09:32.816783 32714 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:32.817065 32655 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEADER]: Committing config change with OpId 4.36: config changed from index 35 to 36, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.817505 31795 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.818135 31717 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.818212 31542 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 reported cstate change: config changed from index 35 to 36, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:32.818135 32051 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } attrs { replace: true } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.819525 32128 consensus_queue.cc:237] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } }
I20251027 03:09:32.819926 32051 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20251027 03:09:32.820143 31795 raft_consensus.cc:1275] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Refusing update from remote peer bcd3eff6ab6f42e9b6b475710ba14c80: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20251027 03:09:32.820123 32714 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.820329 32655 consensus_queue.cc:1048] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:32.820835 32714 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEADER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.820992 31795 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.820992 32051 raft_consensus.cc:2955] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.821924 31538 catalog_manager.cc:5649] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 reported cstate change: config changed from index 36 to 37, VOTER 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) evicted. New cstate: current_term: 4 leader_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } } peers { permanent_uuid: "bcd3eff6ab6f42e9b6b475710ba14c80" member_type: VOTER last_known_addr { host: "127.27.151.69" port: 42233 } attrs { promote: false } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } }
I20251027 03:09:32.821960 31587 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0e2464cf0e9243b7845c98d8cde8528b with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:32.822507 31704 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e2464cf0e9243b7845c98d8cde8528b with delete_type TABLET_DATA_TOMBSTONED (TS 50aa97f11c164e0b9e3520267cf6c85f not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:57710
I20251027 03:09:32.822610 32709 tablet_replica.cc:333] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:32.822665 32709 raft_consensus.cc:2243] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:32.822732 32709 raft_consensus.cc:2272] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:32.822927 32709 ts_tablet_manager.cc:1916] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:32.823768 32709 ts_tablet_manager.cc:1929] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.36
I20251027 03:09:32.823812 32709 log.cc:1199] T 0e2464cf0e9243b7845c98d8cde8528b P 50aa97f11c164e0b9e3520267cf6c85f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/0e2464cf0e9243b7845c98d8cde8528b
I20251027 03:09:32.824074 31529 catalog_manager.cc:4985] TS 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091): tablet 0e2464cf0e9243b7845c98d8cde8528b (table test-workload [id=dee44af73a894cc0aa4f501bdf139afe]) successfully deleted
I20251027 03:09:33.881242 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:33.881726 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:33.881810 31717 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:33.882026 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:33.882104 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:34.362744 32678 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:34.362825 32678 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:34.363014 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43366
I20251027 03:09:34.363077 31639 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:34.363096 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 7 FOLLOWER]: Advancing to term 8
I20251027 03:09:34.365010 31639 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:34.365134 31639 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 8 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:34.365278 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 8 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:34.365298 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 8 candidate_status { last_received { term: 7 index: 34 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:34.365379 31717 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 LEADER]: Stepping down as leader of term 7
I20251027 03:09:34.365396 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 7 FOLLOWER]: Advancing to term 8
I20251027 03:09:34.365399 31717 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:34.365502 31717 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 7.34, Last appended by leader: 34, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:34.365576 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 7 FOLLOWER]: Advancing to term 8
I20251027 03:09:34.366104 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 8.
I20251027 03:09:34.366284 31609 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:34.366325 31717 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 8.
I20251027 03:09:34.366454 32704 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 FOLLOWER]: Leader election won for term 8
I20251027 03:09:34.366649 32704 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:34.366755 32704 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 7.34, Last appended by leader: 34, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:34.367323 31538 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 7 to 8, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 8 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:34.790768 31717 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 8 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 7 index: 34. Preceding OpId from leader: term: 8 index: 35. (index mismatch)
I20251027 03:09:34.791044 32704 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:34.792371 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 8 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 7 index: 34. Preceding OpId from leader: term: 8 index: 35. (index mismatch)
I20251027 03:09:34.792568 32732 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:35.882241 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:35.882673 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6045341ee60c472bbb01e952650a735a"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "93e976cbba5e4213a43042ffe2da42d6"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:35.882758 31639 raft_consensus.cc:606] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Received request to transfer leadership to TS 93e976cbba5e4213a43042ffe2da42d6
I20251027 03:09:35.882934 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:35.882997 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:36.241246 32735 raft_consensus.cc:993] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 93e976cbba5e4213a43042ffe2da42d6 to start an election
I20251027 03:09:36.241320 32735 raft_consensus.cc:1081] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Signalling peer 93e976cbba5e4213a43042ffe2da42d6 to start an election
I20251027 03:09:36.241472 32051 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6045341ee60c472bbb01e952650a735a"
dest_uuid: "93e976cbba5e4213a43042ffe2da42d6"
from {username='slave'} at 127.0.0.1:56712
I20251027 03:09:36.241549 32051 raft_consensus.cc:493] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:36.241568 32051 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:36.242164 32051 raft_consensus.cc:515] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:36.242298 32051 leader_election.cc:290] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [CANDIDATE]: Term 2 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:36.242479 32202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "93e976cbba5e4213a43042ffe2da42d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: true dest_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
I20251027 03:09:36.242508 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "93e976cbba5e4213a43042ffe2da42d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:36.242543 32202 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:36.242558 31639 raft_consensus.cc:3055] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:36.242579 31639 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:36.242631 31639 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 34, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:36.242718 31639 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:36.243184 31639 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e976cbba5e4213a43042ffe2da42d6 in term 2.
I20251027 03:09:36.243192 32202 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93e976cbba5e4213a43042ffe2da42d6 in term 2.
I20251027 03:09:36.243355 32017 leader_election.cc:304] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93e976cbba5e4213a43042ffe2da42d6, d421b303d20a48cfbbd34da50b8d0d6d; no voters:
I20251027 03:09:36.243456 32720 raft_consensus.cc:2804] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:36.243558 32720 raft_consensus.cc:697] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEADER]: Becoming Leader. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Running, Role: LEADER
I20251027 03:09:36.243616 32720 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 1.34, Last appended by leader: 34, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:36.244098 31542 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 reported cstate change: term changed from 1 to 2, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68). New cstate: current_term: 2 leader_uuid: "93e976cbba5e4213a43042ffe2da42d6" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:36.701826 32202 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 2 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20251027 03:09:36.702105 32720 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:36.703238 31639 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Refusing update from remote peer 93e976cbba5e4213a43042ffe2da42d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20251027 03:09:36.703419 32747 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251027 03:09:37.883226 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:37.889803 32051 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6045341ee60c472bbb01e952650a735a"
dest_uuid: "93e976cbba5e4213a43042ffe2da42d6"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:41582
I20251027 03:09:37.889883 32051 raft_consensus.cc:606] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:37.890029 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:37.890093 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:38.229238 32747 raft_consensus.cc:993] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:38.229332 32747 raft_consensus.cc:1081] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:38.229527 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6045341ee60c472bbb01e952650a735a"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43412
I20251027 03:09:38.229609 31639 raft_consensus.cc:493] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:38.229628 31639 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:38.230293 31639 raft_consensus.cc:515] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:38.230454 31639 leader_election.cc:290] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Requested vote from peers 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68:44331), d421b303d20a48cfbbd34da50b8d0d6d (127.27.151.70:42021)
I20251027 03:09:38.230595 32202 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 35 } } ignore_live_leader: true dest_uuid: "d421b303d20a48cfbbd34da50b8d0d6d"
I20251027 03:09:38.230684 32202 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:38.230595 32051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6045341ee60c472bbb01e952650a735a" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 3 candidate_status { last_received { term: 2 index: 35 } } ignore_live_leader: true dest_uuid: "93e976cbba5e4213a43042ffe2da42d6"
I20251027 03:09:38.230736 32051 raft_consensus.cc:3055] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:38.230770 32051 raft_consensus.cc:740] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 93e976cbba5e4213a43042ffe2da42d6, State: Running, Role: LEADER
I20251027 03:09:38.230822 32051 consensus_queue.cc:260] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:38.230897 32051 raft_consensus.cc:3060] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:38.231266 32202 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:38.231448 31611 leader_election.cc:304] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78e34a95a4a44fde91fa13a71fa3da26, d421b303d20a48cfbbd34da50b8d0d6d; no voters:
I20251027 03:09:38.231492 32051 raft_consensus.cc:2468] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 3.
I20251027 03:09:38.231552 32735 raft_consensus.cc:2804] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:38.231662 32735 raft_consensus.cc:697] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:38.231731 32735 consensus_queue.cc:237] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } }
I20251027 03:09:38.232276 31538 catalog_manager.cc:5649] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 2 to 3, leader changed from 93e976cbba5e4213a43042ffe2da42d6 (127.27.151.68) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 3 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:38.749755 32051 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20251027 03:09:38.750002 32735 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e976cbba5e4213a43042ffe2da42d6" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:38.751169 32202 raft_consensus.cc:1275] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 2 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20251027 03:09:38.751320 32762 consensus_queue.cc:1048] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d421b303d20a48cfbbd34da50b8d0d6d" member_type: VOTER last_known_addr { host: "127.27.151.70" port: 42021 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:39.890252 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:39.890688 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:39.890765 31639 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:39.890946 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:39.890995 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:40.010500 32766 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:40.010582 32766 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:40.010763 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:40.010848 31717 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:40.010890 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 8 FOLLOWER]: Advancing to term 9
I20251027 03:09:40.011514 31717 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:40.011633 31717 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 9 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:40.011790 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:40.011848 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 8 FOLLOWER]: Advancing to term 9
I20251027 03:09:40.011803 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:40.011941 31639 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 LEADER]: Stepping down as leader of term 8
I20251027 03:09:40.011973 31639 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:40.012032 31639 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 8.35, Last appended by leader: 35, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:40.012105 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 8 FOLLOWER]: Advancing to term 9
I20251027 03:09:40.012465 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 9.
I20251027 03:09:40.012634 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:40.012686 31639 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 9.
I20251027 03:09:40.012709 32678 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 FOLLOWER]: Leader election won for term 9
I20251027 03:09:40.012809 32678 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:40.012862 32678 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 8.35, Last appended by leader: 35, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:40.013397 31538 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 8 to 9, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 9 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:40.444175 31639 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 9 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 8 index: 35. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20251027 03:09:40.444411 32678 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:40.445489 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 9 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 8 index: 35. Preceding OpId from leader: term: 9 index: 36. (index mismatch)
I20251027 03:09:40.445683 300 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20251027 03:09:41.891232 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:41.891709 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:41.891793 31717 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:41.892019 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:41.892078 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:42.233873 300 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:42.233955 300 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:42.234125 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43366
I20251027 03:09:42.234208 31639 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:42.234229 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 9 FOLLOWER]: Advancing to term 10
I20251027 03:09:42.234813 31639 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:42.234946 31639 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 10 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:42.235066 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:42.235141 31717 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 LEADER]: Stepping down as leader of term 9
I20251027 03:09:42.235131 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 10 candidate_status { last_received { term: 9 index: 36 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:42.235203 31717 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:42.235224 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 9 FOLLOWER]: Advancing to term 10
I20251027 03:09:42.235266 31717 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:42.235370 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 9 FOLLOWER]: Advancing to term 10
I20251027 03:09:42.235812 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 10.
I20251027 03:09:42.235985 31717 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 10.
I20251027 03:09:42.235986 31609 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:42.236086 32766 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 FOLLOWER]: Leader election won for term 10
I20251027 03:09:42.236199 32766 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:42.236266 32766 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 9.36, Last appended by leader: 36, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:42.236750 31538 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 9 to 10, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 10 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:42.692915 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 10 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
I20251027 03:09:42.693164 32766 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:42.694474 31717 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 10 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 9 index: 36. Preceding OpId from leader: term: 10 index: 37. (index mismatch)
I20251027 03:09:42.694654 32766 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20251027 03:09:43.892206 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:43.892585 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:43.892665 31639 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:43.892825 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:43.892879 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:44.183543 323 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:44.183619 323 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:44.183807 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:44.183878 31717 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:44.183897 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 10 FOLLOWER]: Advancing to term 11
I20251027 03:09:44.184525 31717 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:44.184648 31717 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 11 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:44.184760 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 11 candidate_status { last_received { term: 10 index: 37 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:44.184818 31639 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 LEADER]: Stepping down as leader of term 10
I20251027 03:09:44.184796 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 11 candidate_status { last_received { term: 10 index: 37 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:44.184875 31639 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:44.184923 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 10 FOLLOWER]: Advancing to term 11
I20251027 03:09:44.184963 31639 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 10.37, Last appended by leader: 37, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:44.185042 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 10 FOLLOWER]: Advancing to term 11
I20251027 03:09:44.185444 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 11.
I20251027 03:09:44.185510 31639 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 11.
I20251027 03:09:44.185645 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:44.185731 300 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 FOLLOWER]: Leader election won for term 11
I20251027 03:09:44.185838 300 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:44.185906 300 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 10.37, Last appended by leader: 37, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:44.186380 31542 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 10 to 11, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 11 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:44.783529 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 11 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 10 index: 37. Preceding OpId from leader: term: 11 index: 38. (index mismatch)
I20251027 03:09:44.783768 300 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251027 03:09:44.784795 31639 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 11 FOLLOWER]: Refusing update from remote peer 50aa97f11c164e0b9e3520267cf6c85f: Log matching property violated. Preceding OpId in replica: term: 10 index: 37. Preceding OpId from leader: term: 11 index: 38. (index mismatch)
I20251027 03:09:44.785075 300 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20251027 03:09:45.893033 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:45.893453 31717 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
mode: GRACEFUL
new_leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:57750
I20251027 03:09:45.893537 31717 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 LEADER]: Received request to transfer leadership to TS 78e34a95a4a44fde91fa13a71fa3da26
I20251027 03:09:45.893712 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:45.893775 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:45.984123 336 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: : Instructing follower 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:45.984215 336 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 LEADER]: Signalling peer 78e34a95a4a44fde91fa13a71fa3da26 to start an election
I20251027 03:09:45.984409 31639 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
from {username='slave'} at 127.0.0.1:43366
I20251027 03:09:45.984498 31639 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:45.984519 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 11 FOLLOWER]: Advancing to term 12
I20251027 03:09:45.985118 31639 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:45.985275 31639 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 12 election: Requested vote from peers 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66:41091), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:45.985400 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 12 candidate_status { last_received { term: 11 index: 38 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:45.985400 31717 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "78e34a95a4a44fde91fa13a71fa3da26" candidate_term: 12 candidate_status { last_received { term: 11 index: 38 } } ignore_live_leader: true dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
I20251027 03:09:45.985484 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 11 FOLLOWER]: Advancing to term 12
I20251027 03:09:45.985503 31717 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 LEADER]: Stepping down as leader of term 11
I20251027 03:09:45.985531 31717 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:45.985595 31717 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 11.38, Last appended by leader: 38, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:45.985668 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 11 FOLLOWER]: Advancing to term 12
I20251027 03:09:45.986188 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 12.
I20251027 03:09:45.986315 31717 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78e34a95a4a44fde91fa13a71fa3da26 in term 12.
I20251027 03:09:45.986354 31609 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 78e34a95a4a44fde91fa13a71fa3da26; no voters:
I20251027 03:09:45.986469 323 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 FOLLOWER]: Leader election won for term 12
I20251027 03:09:45.986574 323 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 LEADER]: Becoming Leader. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:45.986646 323 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 11.38, Last appended by leader: 38, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:45.987159 31542 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 reported cstate change: term changed from 11 to 12, leader changed from 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66) to 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65). New cstate: current_term: 12 leader_uuid: "78e34a95a4a44fde91fa13a71fa3da26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:46.432857 31795 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 12 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 11 index: 38. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20251027 03:09:46.433435 323 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251027 03:09:46.434765 31717 raft_consensus.cc:1275] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 12 FOLLOWER]: Refusing update from remote peer 78e34a95a4a44fde91fa13a71fa3da26: Log matching property violated. Preceding OpId in replica: term: 11 index: 38. Preceding OpId from leader: term: 12 index: 39. (index mismatch)
I20251027 03:09:46.435005 349 consensus_queue.cc:1048] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20251027 03:09:47.894035 31598 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:47.894516 31639 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
mode: GRACEFUL
new_leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:43398
I20251027 03:09:47.894590 31639 raft_consensus.cc:606] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 LEADER]: Received request to transfer leadership to TS 50aa97f11c164e0b9e3520267cf6c85f
I20251027 03:09:47.894784 31598 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:47.894830 31598 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:48.127079 349 raft_consensus.cc:993] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: : Instructing follower 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:48.127162 349 raft_consensus.cc:1081] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 LEADER]: Signalling peer 50aa97f11c164e0b9e3520267cf6c85f to start an election
I20251027 03:09:48.127344 31717 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6"
dest_uuid: "50aa97f11c164e0b9e3520267cf6c85f"
from {username='slave'} at 127.0.0.1:57716
I20251027 03:09:48.127442 31717 raft_consensus.cc:493] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:48.127472 31717 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 12 FOLLOWER]: Advancing to term 13
I20251027 03:09:48.128140 31717 raft_consensus.cc:515] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:48.128365 31717 leader_election.cc:290] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 13 election: Requested vote from peers 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65:32889), 10153e76038242fdaec0d401cdbb0411 (127.27.151.67:37045)
I20251027 03:09:48.128525 31795 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 13 candidate_status { last_received { term: 12 index: 39 } } ignore_live_leader: true dest_uuid: "10153e76038242fdaec0d401cdbb0411"
I20251027 03:09:48.128609 31795 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 12 FOLLOWER]: Advancing to term 13
I20251027 03:09:48.129168 31795 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 13.
I20251027 03:09:48.129262 31639 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42bb6d5c4d0e42af9c337417582580b6" candidate_uuid: "50aa97f11c164e0b9e3520267cf6c85f" candidate_term: 13 candidate_status { last_received { term: 12 index: 39 } } ignore_live_leader: true dest_uuid: "78e34a95a4a44fde91fa13a71fa3da26"
I20251027 03:09:48.129334 31639 raft_consensus.cc:3055] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 LEADER]: Stepping down as leader of term 12
I20251027 03:09:48.129356 31685 leader_election.cc:304] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10153e76038242fdaec0d401cdbb0411, 50aa97f11c164e0b9e3520267cf6c85f; no voters:
I20251027 03:09:48.129366 31639 raft_consensus.cc:740] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 78e34a95a4a44fde91fa13a71fa3da26, State: Running, Role: LEADER
I20251027 03:09:48.129469 336 raft_consensus.cc:2804] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 13 FOLLOWER]: Leader election won for term 13
I20251027 03:09:48.129478 31639 consensus_queue.cc:260] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 12.39, Last appended by leader: 39, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:48.129572 336 raft_consensus.cc:697] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 13 LEADER]: Becoming Leader. State: Replica: 50aa97f11c164e0b9e3520267cf6c85f, State: Running, Role: LEADER
I20251027 03:09:48.129618 31639 raft_consensus.cc:3060] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 12 FOLLOWER]: Advancing to term 13
I20251027 03:09:48.129619 336 consensus_queue.cc:237] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 12.39, Last appended by leader: 39, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } }
I20251027 03:09:48.130123 31542 catalog_manager.cc:5649] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f reported cstate change: term changed from 12 to 13, leader changed from 78e34a95a4a44fde91fa13a71fa3da26 (127.27.151.65) to 50aa97f11c164e0b9e3520267cf6c85f (127.27.151.66). New cstate: current_term: 13 leader_uuid: "50aa97f11c164e0b9e3520267cf6c85f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78e34a95a4a44fde91fa13a71fa3da26" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 32889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50aa97f11c164e0b9e3520267cf6c85f" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10153e76038242fdaec0d401cdbb0411" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 37045 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:48.130263 31639 raft_consensus.cc:2468] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50aa97f11c164e0b9e3520267cf6c85f in term 13.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:196: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20251027 03:09:48.361860 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:09:48.365113 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:48.365226 28253 tablet_replica.cc:333] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:48.365295 28253 raft_consensus.cc:2243] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.365362 28253 raft_consensus.cc:2272] T 6045341ee60c472bbb01e952650a735a P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.365542 28253 tablet_replica.cc:333] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:48.365576 28253 raft_consensus.cc:2243] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.365620 28253 raft_consensus.cc:2272] T 914edf6a5ed5402fb1d07b6afba645b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.365789 28253 tablet_replica.cc:333] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:48.365823 28253 raft_consensus.cc:2243] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 13 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.365856 28253 raft_consensus.cc:2272] T 42bb6d5c4d0e42af9c337417582580b6 P 78e34a95a4a44fde91fa13a71fa3da26 [term 13 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.365967 28253 tablet_replica.cc:333] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:48.365994 28253 raft_consensus.cc:2243] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.366070 28253 raft_consensus.cc:2272] T 645f3c62551e42109ebfe785bde36416 P 78e34a95a4a44fde91fa13a71fa3da26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.366175 28253 tablet_replica.cc:333] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:48.366202 28253 raft_consensus.cc:2243] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.366262 28253 raft_consensus.cc:2272] T 30ef3ff09aa247fab2eae47a3f1718fd P 78e34a95a4a44fde91fa13a71fa3da26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.366374 28253 tablet_replica.cc:333] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26: stopping tablet replica
I20251027 03:09:48.366402 28253 raft_consensus.cc:2243] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.366487 28253 raft_consensus.cc:2272] T 5e1ec5a8f7224329be0fe0e5453d71fc P 78e34a95a4a44fde91fa13a71fa3da26 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.378954 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:09:48.380240 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:09:48.383553 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:48.383651 28253 tablet_replica.cc:333] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:48.383708 28253 raft_consensus.cc:2243] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.383769 28253 raft_consensus.cc:2272] T c176d9f872b84ac986522990949395eb P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.383872 28253 tablet_replica.cc:333] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:48.383903 28253 raft_consensus.cc:2243] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.383932 28253 raft_consensus.cc:2272] T bb4aebbafbe6415e90eaf6f6feb6b760 P 50aa97f11c164e0b9e3520267cf6c85f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.384011 28253 tablet_replica.cc:333] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:48.384035 28253 raft_consensus.cc:2243] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.384056 28253 raft_consensus.cc:2272] T 726dd98dfd6c47a5b95b723de71cb5c5 P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.384124 28253 tablet_replica.cc:333] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:48.384200 28253 raft_consensus.cc:2243] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.384250 28253 raft_consensus.cc:2272] T 11ce7fc7b4334ffe99df2944dca2af4a P 50aa97f11c164e0b9e3520267cf6c85f [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.384332 28253 tablet_replica.cc:333] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:48.384356 28253 raft_consensus.cc:2243] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 13 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.384385 28253 pending_rounds.cc:70] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: Trying to abort 1 pending ops.
I20251027 03:09:48.384402 28253 pending_rounds.cc:77] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f: Aborting op as it isn't in flight: id { term: 13 index: 40 } timestamp: 7215245672979427328 op_type: NO_OP noop_request { }
I20251027 03:09:48.384438 28253 raft_consensus.cc:2889] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 13 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:09:48.384459 28253 raft_consensus.cc:2272] T 42bb6d5c4d0e42af9c337417582580b6 P 50aa97f11c164e0b9e3520267cf6c85f [term 13 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.384598 28253 tablet_replica.cc:333] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f: stopping tablet replica
I20251027 03:09:48.384632 28253 raft_consensus.cc:2243] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.384673 28253 raft_consensus.cc:2272] T a7a4ca641cf349f29eeb0292a996449a P 50aa97f11c164e0b9e3520267cf6c85f [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.396471 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:09:48.397742 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:09:48.400724 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:48.400821 28253 tablet_replica.cc:333] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:48.400862 28253 raft_consensus.cc:2243] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.400919 28253 raft_consensus.cc:2272] T 914edf6a5ed5402fb1d07b6afba645b6 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.401043 28253 tablet_replica.cc:333] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:48.401082 28253 raft_consensus.cc:2243] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.401151 28253 raft_consensus.cc:2272] T bb4aebbafbe6415e90eaf6f6feb6b760 P 10153e76038242fdaec0d401cdbb0411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.401258 28253 tablet_replica.cc:333] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:48.401290 28253 raft_consensus.cc:2243] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.401340 28253 raft_consensus.cc:2272] T 0e2464cf0e9243b7845c98d8cde8528b P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.401453 28253 tablet_replica.cc:333] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:48.401484 28253 raft_consensus.cc:2243] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 5 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.401543 28253 raft_consensus.cc:2272] T 726dd98dfd6c47a5b95b723de71cb5c5 P 10153e76038242fdaec0d401cdbb0411 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.401683 28253 tablet_replica.cc:333] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:48.401717 28253 raft_consensus.cc:2243] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 13 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.401757 28253 raft_consensus.cc:2272] T 42bb6d5c4d0e42af9c337417582580b6 P 10153e76038242fdaec0d401cdbb0411 [term 13 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.401876 28253 tablet_replica.cc:333] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411: stopping tablet replica
I20251027 03:09:48.401902 28253 raft_consensus.cc:2243] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.401933 28253 raft_consensus.cc:2272] T a7a4ca641cf349f29eeb0292a996449a P 10153e76038242fdaec0d401cdbb0411 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.414342 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:09:48.415591 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:09:48.417671 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:48.417760 28253 tablet_replica.cc:333] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6: stopping tablet replica
I20251027 03:09:48.417807 28253 raft_consensus.cc:2243] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.417862 28253 raft_consensus.cc:2272] T 0e2464cf0e9243b7845c98d8cde8528b P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.417984 28253 tablet_replica.cc:333] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6: stopping tablet replica
I20251027 03:09:48.418027 28253 raft_consensus.cc:2243] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.418081 28253 raft_consensus.cc:2272] T 11ce7fc7b4334ffe99df2944dca2af4a P 93e976cbba5e4213a43042ffe2da42d6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.418184 28253 tablet_replica.cc:333] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6: stopping tablet replica
I20251027 03:09:48.418218 28253 raft_consensus.cc:2243] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.418262 28253 raft_consensus.cc:2272] T 645f3c62551e42109ebfe785bde36416 P 93e976cbba5e4213a43042ffe2da42d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.418365 28253 tablet_replica.cc:333] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6: stopping tablet replica
I20251027 03:09:48.418397 28253 raft_consensus.cc:2243] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.418431 28253 raft_consensus.cc:2272] T 6045341ee60c472bbb01e952650a735a P 93e976cbba5e4213a43042ffe2da42d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.418532 28253 tablet_replica.cc:333] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6: stopping tablet replica
I20251027 03:09:48.418565 28253 raft_consensus.cc:2243] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.418614 28253 raft_consensus.cc:2272] T a7a4ca641cf349f29eeb0292a996449a P 93e976cbba5e4213a43042ffe2da42d6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.418725 28253 tablet_replica.cc:333] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6: stopping tablet replica
I20251027 03:09:48.418756 28253 raft_consensus.cc:2243] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.418816 28253 raft_consensus.cc:2272] T c176d9f872b84ac986522990949395eb P 93e976cbba5e4213a43042ffe2da42d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.430374 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:09:48.431388 28253 tablet_server.cc:178] TabletServer@127.27.151.69:0 shutting down...
I20251027 03:09:48.433554 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:48.433681 28253 tablet_replica.cc:333] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80: stopping tablet replica
I20251027 03:09:48.433743 28253 raft_consensus.cc:2243] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.433795 28253 raft_consensus.cc:2272] T 5e1ec5a8f7224329be0fe0e5453d71fc P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.433926 28253 tablet_replica.cc:333] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80: stopping tablet replica
I20251027 03:09:48.433969 28253 raft_consensus.cc:2243] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.434017 28253 raft_consensus.cc:2272] T 11ce7fc7b4334ffe99df2944dca2af4a P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.434156 28253 tablet_replica.cc:333] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80: stopping tablet replica
I20251027 03:09:48.434192 28253 raft_consensus.cc:2243] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.434252 28253 raft_consensus.cc:2272] T 30ef3ff09aa247fab2eae47a3f1718fd P bcd3eff6ab6f42e9b6b475710ba14c80 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.434372 28253 tablet_replica.cc:333] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80: stopping tablet replica
I20251027 03:09:48.434422 28253 raft_consensus.cc:2243] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.434480 28253 raft_consensus.cc:2272] T bb4aebbafbe6415e90eaf6f6feb6b760 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.434582 28253 tablet_replica.cc:333] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80: stopping tablet replica
I20251027 03:09:48.434615 28253 raft_consensus.cc:2243] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.434688 28253 raft_consensus.cc:2272] T 0e2464cf0e9243b7845c98d8cde8528b P bcd3eff6ab6f42e9b6b475710ba14c80 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.434800 28253 tablet_replica.cc:333] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80: stopping tablet replica
I20251027 03:09:48.434832 28253 raft_consensus.cc:2243] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.434881 28253 raft_consensus.cc:2272] T 726dd98dfd6c47a5b95b723de71cb5c5 P bcd3eff6ab6f42e9b6b475710ba14c80 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.446331 28253 tablet_server.cc:195] TabletServer@127.27.151.69:0 shutdown complete.
I20251027 03:09:48.447348 28253 tablet_server.cc:178] TabletServer@127.27.151.70:0 shutting down...
I20251027 03:09:48.449846 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:48.449957 28253 tablet_replica.cc:333] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d: stopping tablet replica
I20251027 03:09:48.450011 28253 raft_consensus.cc:2243] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.450063 28253 raft_consensus.cc:2272] T 30ef3ff09aa247fab2eae47a3f1718fd P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.450189 28253 tablet_replica.cc:333] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d: stopping tablet replica
I20251027 03:09:48.450227 28253 raft_consensus.cc:2243] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.450273 28253 raft_consensus.cc:2272] T c176d9f872b84ac986522990949395eb P d421b303d20a48cfbbd34da50b8d0d6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.450382 28253 tablet_replica.cc:333] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d: stopping tablet replica
I20251027 03:09:48.450416 28253 raft_consensus.cc:2243] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.450466 28253 raft_consensus.cc:2272] T 5e1ec5a8f7224329be0fe0e5453d71fc P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.450568 28253 tablet_replica.cc:333] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d: stopping tablet replica
I20251027 03:09:48.450603 28253 raft_consensus.cc:2243] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.450642 28253 raft_consensus.cc:2272] T 914edf6a5ed5402fb1d07b6afba645b6 P d421b303d20a48cfbbd34da50b8d0d6d [term 5 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.450744 28253 tablet_replica.cc:333] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d: stopping tablet replica
I20251027 03:09:48.450775 28253 raft_consensus.cc:2243] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.450829 28253 raft_consensus.cc:2272] T 645f3c62551e42109ebfe785bde36416 P d421b303d20a48cfbbd34da50b8d0d6d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.450932 28253 tablet_replica.cc:333] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d: stopping tablet replica
I20251027 03:09:48.450968 28253 raft_consensus.cc:2243] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:48.451012 28253 raft_consensus.cc:2272] T 6045341ee60c472bbb01e952650a735a P d421b303d20a48cfbbd34da50b8d0d6d [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.462689 28253 tablet_server.cc:195] TabletServer@127.27.151.70:0 shutdown complete.
I20251027 03:09:48.463617 28253 master.cc:561] Master@127.27.151.126:37795 shutting down...
I20251027 03:09:48.467079 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:48.467136 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:48.467168 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P c6144ee9e79f41fc948fc9c34a5a33e0: stopping tablet replica
I20251027 03:09:48.478876 28253 master.cc:583] Master@127.27.151.126:37795 shutdown complete.
I20251027 03:09:48.480659 28253 test_util.cc:183] -----------------------------------------------
I20251027 03:09:48.480705 28253 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761534461946371-28253-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (34648 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20251027 03:09:48.481149 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:43747
I20251027 03:09:48.481288 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:48.482507 365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:48.482547 363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:48.482725 362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:48.482755 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:48.482857 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:48.482893 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:48.482909 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534588482909 us; error 0 us; skew 500 ppm
I20251027 03:09:48.483518 28253 webserver.cc:492] Webserver started at http://127.27.151.126:38027/ using document root <none> and password file <none>
I20251027 03:09:48.483604 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:48.483651 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:48.483701 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:48.483944 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "50e215f477fb4cf2b0506402920c88ba"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.484869 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:48.485304 370 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.485422 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.485466 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "50e215f477fb4cf2b0506402920c88ba"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.485514 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:48.496909 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:48.497090 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:48.499325 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:43747
I20251027 03:09:48.505932 422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:43747 every 8 connection(s)
I20251027 03:09:48.506064 423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:48.506960 423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba: Bootstrap starting.
I20251027 03:09:48.507210 423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:48.507663 423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba: No bootstrap required, opened a new log
I20251027 03:09:48.507813 423 raft_consensus.cc:359] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e215f477fb4cf2b0506402920c88ba" member_type: VOTER }
I20251027 03:09:48.507861 423 raft_consensus.cc:385] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:48.507875 423 raft_consensus.cc:740] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e215f477fb4cf2b0506402920c88ba, State: Initialized, Role: FOLLOWER
I20251027 03:09:48.507918 423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e215f477fb4cf2b0506402920c88ba" member_type: VOTER }
I20251027 03:09:48.507954 423 raft_consensus.cc:399] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:09:48.507973 423 raft_consensus.cc:493] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:09:48.508000 423 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:48.508481 423 raft_consensus.cc:515] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e215f477fb4cf2b0506402920c88ba" member_type: VOTER }
I20251027 03:09:48.508533 423 leader_election.cc:304] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50e215f477fb4cf2b0506402920c88ba; no voters:
I20251027 03:09:48.508628 423 leader_election.cc:290] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:09:48.508678 426 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:48.508772 423 sys_catalog.cc:565] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:09:48.508786 426 raft_consensus.cc:697] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 1 LEADER]: Becoming Leader. State: Replica: 50e215f477fb4cf2b0506402920c88ba, State: Running, Role: LEADER
I20251027 03:09:48.508837 426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e215f477fb4cf2b0506402920c88ba" member_type: VOTER }
I20251027 03:09:48.509061 427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50e215f477fb4cf2b0506402920c88ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e215f477fb4cf2b0506402920c88ba" member_type: VOTER } }
I20251027 03:09:48.509117 427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:48.509095 428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50e215f477fb4cf2b0506402920c88ba. Latest consensus state: current_term: 1 leader_uuid: "50e215f477fb4cf2b0506402920c88ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50e215f477fb4cf2b0506402920c88ba" member_type: VOTER } }
I20251027 03:09:48.509152 428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:48.509322 431 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:09:48.509461 431 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:09:48.509945 431 catalog_manager.cc:1357] Generated new cluster ID: d8241e593b534023abbab8ba1c0edeb8
I20251027 03:09:48.509987 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:09:48.510006 431 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:09:48.520381 431 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:09:48.520740 431 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:09:48.524121 431 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba: Generated new TSK 0
I20251027 03:09:48.524205 431 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:09:48.525481 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:48.526594 444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:48.526659 445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:48.526701 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:48.526683 447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:48.526868 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:48.526914 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:48.526934 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534588526934 us; error 0 us; skew 500 ppm
I20251027 03:09:48.527451 28253 webserver.cc:492] Webserver started at http://127.27.151.65:35549/ using document root <none> and password file <none>
I20251027 03:09:48.527532 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:48.527577 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:48.527623 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:48.527863 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "84aaafc3085e46c385072b38ef11da96"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.528674 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:48.529079 453 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.529187 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.529225 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "84aaafc3085e46c385072b38ef11da96"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.529268 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:48.553236 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:48.553392 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:48.553517 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:48.553691 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:48.553716 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.553747 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:48.553774 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.556397 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:43553
I20251027 03:09:48.556442 516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:43553 every 8 connection(s)
I20251027 03:09:48.556761 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:48.557794 521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:48.557803 524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:48.557947 522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:48.558151 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:48.558251 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:48.558277 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:48.558300 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534588558299 us; error 0 us; skew 500 ppm
I20251027 03:09:48.558784 28253 webserver.cc:492] Webserver started at http://127.27.151.66:39331/ using document root <none> and password file <none>
I20251027 03:09:48.558867 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:48.558905 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:48.558951 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:48.559180 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.560029 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:48.560434 529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.560560 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.560608 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.560667 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:48.563285 517 heartbeater.cc:344] Connected to a master server at 127.27.151.126:43747
I20251027 03:09:48.563344 517 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:48.563433 517 heartbeater.cc:507] Master 127.27.151.126:43747 requested a full tablet report, sending...
I20251027 03:09:48.563643 387 ts_manager.cc:194] Registered new tserver with Master: 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553)
I20251027 03:09:48.564229 387 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42676
I20251027 03:09:48.581164 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:48.581321 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:48.581426 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:48.581550 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:48.581574 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.581602 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:48.581619 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.584193 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:45827
I20251027 03:09:48.584252 593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:45827 every 8 connection(s)
I20251027 03:09:48.584515 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:48.585695 600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:48.585718 597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:48.585901 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:48.586334 598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:48.586524 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:48.586601 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:48.586632 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534588586632 us; error 0 us; skew 500 ppm
I20251027 03:09:48.587255 28253 webserver.cc:492] Webserver started at http://127.27.151.67:36593/ using document root <none> and password file <none>
I20251027 03:09:48.587388 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:48.587431 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:48.587481 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:48.587507 594 heartbeater.cc:344] Connected to a master server at 127.27.151.126:43747
I20251027 03:09:48.587639 594 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:48.587762 594 heartbeater.cc:507] Master 127.27.151.126:43747 requested a full tablet report, sending...
I20251027 03:09:48.587864 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.588034 387 ts_manager.cc:194] Registered new tserver with Master: a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827)
I20251027 03:09:48.588510 387 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42682
I20251027 03:09:48.589052 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:48.589438 605 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.589550 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.589584 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
format_stamp: "Formatted at 2025-10-27 03:09:48 on dist-test-slave-4cc8"
I20251027 03:09:48.589627 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:48.597792 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:48.597913 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:48.598024 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:48.598168 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:48.598192 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.598209 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:48.598237 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:48.600739 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:35649
I20251027 03:09:48.600785 668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:35649 every 8 connection(s)
I20251027 03:09:48.603233 669 heartbeater.cc:344] Connected to a master server at 127.27.151.126:43747
I20251027 03:09:48.603281 669 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:48.603343 669 heartbeater.cc:507] Master 127.27.151.126:43747 requested a full tablet report, sending...
I20251027 03:09:48.603514 387 ts_manager.cc:194] Registered new tserver with Master: 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649)
I20251027 03:09:48.603821 387 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42694
I20251027 03:09:48.604163 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003288073s
I20251027 03:09:49.564812 517 heartbeater.cc:499] Master 127.27.151.126:43747 was elected leader, sending a full tablet report...
I20251027 03:09:49.589010 594 heartbeater.cc:499] Master 127.27.151.126:43747 was elected leader, sending a full tablet report...
I20251027 03:09:49.604209 669 heartbeater.cc:499] Master 127.27.151.126:43747 was elected leader, sending a full tablet report...
I20251027 03:09:49.628918 28253 test_util.cc:276] Using random seed: 997814773
I20251027 03:09:49.633499 387 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42700:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:09:49.633667 387 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:09:49.638561 558 tablet_service.cc:1505] Processing CreateTablet for tablet 127bb95bfdee44c9a793316e85dac2fd (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251027 03:09:49.638618 481 tablet_service.cc:1505] Processing CreateTablet for tablet 127bb95bfdee44c9a793316e85dac2fd (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251027 03:09:49.638682 558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127bb95bfdee44c9a793316e85dac2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.638738 480 tablet_service.cc:1505] Processing CreateTablet for tablet e270b30eae2f4e079afe00094605a736 (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251027 03:09:49.638829 479 tablet_service.cc:1505] Processing CreateTablet for tablet dd03008069ea40cab1a3bec316426350 (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251027 03:09:49.638866 633 tablet_service.cc:1505] Processing CreateTablet for tablet 127bb95bfdee44c9a793316e85dac2fd (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251027 03:09:49.638919 631 tablet_service.cc:1505] Processing CreateTablet for tablet dd03008069ea40cab1a3bec316426350 (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251027 03:09:49.638986 633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127bb95bfdee44c9a793316e85dac2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.638697 481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127bb95bfdee44c9a793316e85dac2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.639109 480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270b30eae2f4e079afe00094605a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.639325 479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd03008069ea40cab1a3bec316426350. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.640082 557 tablet_service.cc:1505] Processing CreateTablet for tablet e270b30eae2f4e079afe00094605a736 (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251027 03:09:49.638862 632 tablet_service.cc:1505] Processing CreateTablet for tablet e270b30eae2f4e079afe00094605a736 (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251027 03:09:49.640213 556 tablet_service.cc:1505] Processing CreateTablet for tablet dd03008069ea40cab1a3bec316426350 (DEFAULT_TABLE table=test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251027 03:09:49.640275 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270b30eae2f4e079afe00094605a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.640432 631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd03008069ea40cab1a3bec316426350. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.640650 691 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: Bootstrap starting.
I20251027 03:09:49.640748 690 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Bootstrap starting.
I20251027 03:09:49.640178 557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270b30eae2f4e079afe00094605a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.641067 556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd03008069ea40cab1a3bec316426350. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:49.641108 691 tablet_bootstrap.cc:654] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.641698 691 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: No bootstrap required, opened a new log
I20251027 03:09:49.641754 691 ts_tablet_manager.cc:1403] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:49.641961 691 raft_consensus.cc:359] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.642045 691 raft_consensus.cc:385] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.642081 691 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aaafc3085e46c385072b38ef11da96, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.642164 691 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.642343 691 ts_tablet_manager.cc:1434] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.642618 691 tablet_bootstrap.cc:492] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: Bootstrap starting.
I20251027 03:09:49.642813 690 tablet_bootstrap.cc:654] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.643165 691 tablet_bootstrap.cc:654] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.643362 696 raft_consensus.cc:493] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:49.643204 695 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: Bootstrap starting.
I20251027 03:09:49.643411 696 raft_consensus.cc:515] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.643819 695 tablet_bootstrap.cc:654] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.643945 691 tablet_bootstrap.cc:492] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: No bootstrap required, opened a new log
I20251027 03:09:49.643989 691 ts_tablet_manager.cc:1403] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:49.644119 691 raft_consensus.cc:359] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.644199 691 raft_consensus.cc:385] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.644224 691 raft_consensus.cc:740] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aaafc3085e46c385072b38ef11da96, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.644255 691 consensus_queue.cc:260] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.644326 691 ts_tablet_manager.cc:1434] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.644376 691 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: Bootstrap starting.
I20251027 03:09:49.644430 696 leader_election.cc:290] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649), a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827)
I20251027 03:09:49.644749 691 tablet_bootstrap.cc:654] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.645243 695 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: No bootstrap required, opened a new log
I20251027 03:09:49.645289 695 ts_tablet_manager.cc:1403] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:09:49.645371 690 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: No bootstrap required, opened a new log
I20251027 03:09:49.645397 690 ts_tablet_manager.cc:1403] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20251027 03:09:49.645413 695 raft_consensus.cc:359] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.645467 695 raft_consensus.cc:385] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.645483 695 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbed8d5d5e74d6095dba4ad8c5d8099, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.645501 690 raft_consensus.cc:359] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.645540 690 raft_consensus.cc:385] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.645521 695 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.645560 690 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b7b3f419fb49ac988d86079dbb7c31, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.645599 690 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.645650 695 ts_tablet_manager.cc:1434] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.645709 695 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Bootstrap starting.
I20251027 03:09:49.645736 690 ts_tablet_manager.cc:1434] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.645779 690 tablet_bootstrap.cc:492] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31: Bootstrap starting.
I20251027 03:09:49.646111 690 tablet_bootstrap.cc:654] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.646081 695 tablet_bootstrap.cc:654] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.647047 691 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: No bootstrap required, opened a new log
I20251027 03:09:49.647094 691 ts_tablet_manager.cc:1403] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:09:49.647254 691 raft_consensus.cc:359] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.647329 691 raft_consensus.cc:385] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.647348 691 raft_consensus.cc:740] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aaafc3085e46c385072b38ef11da96, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.647357 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" is_pre_election: true
I20251027 03:09:49.647387 691 consensus_queue.cc:260] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.647410 568 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 0.
I20251027 03:09:49.647418 643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" is_pre_election: true
I20251027 03:09:49.647517 643 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 0.
I20251027 03:09:49.647724 455 leader_election.cc:304] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aaafc3085e46c385072b38ef11da96, 9bbed8d5d5e74d6095dba4ad8c5d8099; no voters:
I20251027 03:09:49.647761 690 tablet_bootstrap.cc:492] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31: No bootstrap required, opened a new log
I20251027 03:09:49.647794 690 ts_tablet_manager.cc:1403] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:09:49.647842 696 raft_consensus.cc:2804] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:49.647884 696 raft_consensus.cc:493] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:49.647909 696 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.647936 690 raft_consensus.cc:359] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.648000 690 raft_consensus.cc:385] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.648026 690 raft_consensus.cc:740] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b7b3f419fb49ac988d86079dbb7c31, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.648068 690 consensus_queue.cc:260] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.648216 690 ts_tablet_manager.cc:1434] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.648288 690 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31: Bootstrap starting.
I20251027 03:09:49.648622 690 tablet_bootstrap.cc:654] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.647469 691 ts_tablet_manager.cc:1434] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.648623 696 raft_consensus.cc:515] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.648854 696 leader_election.cc:290] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 election: Requested vote from peers 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649), a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827)
I20251027 03:09:49.648906 695 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: No bootstrap required, opened a new log
I20251027 03:09:49.648942 695 ts_tablet_manager.cc:1403] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:09:49.649029 643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
I20251027 03:09:49.649075 695 raft_consensus.cc:359] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.649142 695 raft_consensus.cc:385] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.649067 643 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.649147 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
I20251027 03:09:49.649207 568 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.649745 643 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 1.
I20251027 03:09:49.649745 568 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 1.
I20251027 03:09:49.649921 455 leader_election.cc:304] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aaafc3085e46c385072b38ef11da96, a0b7b3f419fb49ac988d86079dbb7c31; no voters:
I20251027 03:09:49.649174 695 raft_consensus.cc:740] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbed8d5d5e74d6095dba4ad8c5d8099, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.650029 696 raft_consensus.cc:2804] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:49.650094 690 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31: No bootstrap required, opened a new log
I20251027 03:09:49.650077 696 raft_consensus.cc:697] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Becoming Leader. State: Replica: 84aaafc3085e46c385072b38ef11da96, State: Running, Role: LEADER
I20251027 03:09:49.650085 695 consensus_queue.cc:260] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.650152 690 ts_tablet_manager.cc:1403] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:49.650185 696 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } }
I20251027 03:09:49.650220 695 ts_tablet_manager.cc:1434] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:49.650266 695 tablet_bootstrap.cc:492] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Bootstrap starting.
I20251027 03:09:49.650305 690 raft_consensus.cc:359] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.650368 690 raft_consensus.cc:385] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.650383 690 raft_consensus.cc:740] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b7b3f419fb49ac988d86079dbb7c31, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.650415 690 consensus_queue.cc:260] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.650493 690 ts_tablet_manager.cc:1434] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.650645 695 tablet_bootstrap.cc:654] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:49.651136 387 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84aaafc3085e46c385072b38ef11da96 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "84aaafc3085e46c385072b38ef11da96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:49.651438 695 tablet_bootstrap.cc:492] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099: No bootstrap required, opened a new log
I20251027 03:09:49.651472 695 ts_tablet_manager.cc:1403] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:49.651594 695 raft_consensus.cc:359] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.651654 695 raft_consensus.cc:385] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:49.651684 695 raft_consensus.cc:740] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbed8d5d5e74d6095dba4ad8c5d8099, State: Initialized, Role: FOLLOWER
I20251027 03:09:49.651732 695 consensus_queue.cc:260] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.651816 695 ts_tablet_manager.cc:1434] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.683418 693 raft_consensus.cc:493] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:49.683471 693 raft_consensus.cc:515] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.683548 693 leader_election.cc:290] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827), 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649)
I20251027 03:09:49.683696 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e270b30eae2f4e079afe00094605a736" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" is_pre_election: true
I20251027 03:09:49.683754 643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e270b30eae2f4e079afe00094605a736" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" is_pre_election: true
I20251027 03:09:49.683769 568 raft_consensus.cc:2468] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 0.
I20251027 03:09:49.683804 643 raft_consensus.cc:2468] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 0.
I20251027 03:09:49.683912 455 leader_election.cc:304] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aaafc3085e46c385072b38ef11da96, a0b7b3f419fb49ac988d86079dbb7c31; no voters:
I20251027 03:09:49.683995 693 raft_consensus.cc:2804] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:49.684033 693 raft_consensus.cc:493] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:49.684047 693 raft_consensus.cc:3060] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.684542 693 raft_consensus.cc:515] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.684661 693 leader_election.cc:290] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 election: Requested vote from peers a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827), 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649)
I20251027 03:09:49.684787 643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e270b30eae2f4e079afe00094605a736" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
I20251027 03:09:49.684787 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e270b30eae2f4e079afe00094605a736" candidate_uuid: "84aaafc3085e46c385072b38ef11da96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
I20251027 03:09:49.684876 643 raft_consensus.cc:3060] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.684880 568 raft_consensus.cc:3060] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.685489 643 raft_consensus.cc:2468] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 1.
I20251027 03:09:49.685489 568 raft_consensus.cc:2468] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84aaafc3085e46c385072b38ef11da96 in term 1.
I20251027 03:09:49.685675 455 leader_election.cc:304] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aaafc3085e46c385072b38ef11da96, 9bbed8d5d5e74d6095dba4ad8c5d8099; no voters:
I20251027 03:09:49.685753 693 raft_consensus.cc:2804] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:49.685823 693 raft_consensus.cc:697] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Becoming Leader. State: Replica: 84aaafc3085e46c385072b38ef11da96, State: Running, Role: LEADER
I20251027 03:09:49.685863 693 consensus_queue.cc:237] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.686342 387 catalog_manager.cc:5649] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84aaafc3085e46c385072b38ef11da96 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "84aaafc3085e46c385072b38ef11da96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:49.704895 702 raft_consensus.cc:493] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:09:49.704957 702 raft_consensus.cc:515] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.705089 702 leader_election.cc:290] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553), a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827)
I20251027 03:09:49.707556 491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd03008069ea40cab1a3bec316426350" candidate_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84aaafc3085e46c385072b38ef11da96" is_pre_election: true
I20251027 03:09:49.707557 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd03008069ea40cab1a3bec316426350" candidate_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" is_pre_election: true
I20251027 03:09:49.707657 491 raft_consensus.cc:2468] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bbed8d5d5e74d6095dba4ad8c5d8099 in term 0.
I20251027 03:09:49.707660 568 raft_consensus.cc:2468] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bbed8d5d5e74d6095dba4ad8c5d8099 in term 0.
I20251027 03:09:49.707816 607 leader_election.cc:304] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aaafc3085e46c385072b38ef11da96, 9bbed8d5d5e74d6095dba4ad8c5d8099; no voters:
I20251027 03:09:49.707934 702 raft_consensus.cc:2804] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:09:49.707978 702 raft_consensus.cc:493] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:09:49.707993 702 raft_consensus.cc:3060] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.708540 702 raft_consensus.cc:515] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.708642 702 leader_election.cc:290] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [CANDIDATE]: Term 1 election: Requested vote from peers 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553), a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827)
I20251027 03:09:49.708792 491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd03008069ea40cab1a3bec316426350" candidate_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84aaafc3085e46c385072b38ef11da96"
I20251027 03:09:49.708850 491 raft_consensus.cc:3060] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.708797 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd03008069ea40cab1a3bec316426350" candidate_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
I20251027 03:09:49.708899 568 raft_consensus.cc:3060] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:49.709398 568 raft_consensus.cc:2468] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbed8d5d5e74d6095dba4ad8c5d8099 in term 1.
I20251027 03:09:49.709411 491 raft_consensus.cc:2468] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbed8d5d5e74d6095dba4ad8c5d8099 in term 1.
I20251027 03:09:49.709543 607 leader_election.cc:304] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bbed8d5d5e74d6095dba4ad8c5d8099, a0b7b3f419fb49ac988d86079dbb7c31; no voters:
I20251027 03:09:49.709626 702 raft_consensus.cc:2804] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:49.709776 702 raft_consensus.cc:697] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 LEADER]: Becoming Leader. State: Replica: 9bbed8d5d5e74d6095dba4ad8c5d8099, State: Running, Role: LEADER
I20251027 03:09:49.709854 702 consensus_queue.cc:237] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } }
I20251027 03:09:49.710335 387 catalog_manager.cc:5649] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } health_report { overall_health: HEALTHY } } }
I20251027 03:09:49.728731 693 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:49.729070 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:49.729693 693 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20251027 03:09:49.729738 693 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20251027 03:09:49.730530 717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:49.730592 716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:49.730749 719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:49.730790 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:49.730963 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:49.730996 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:49.731019 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534589731018 us; error 0 us; skew 500 ppm
I20251027 03:09:49.731699 28253 webserver.cc:492] Webserver started at http://127.27.151.68:33135/ using document root <none> and password file <none>
I20251027 03:09:49.731786 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:49.731827 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:49.731870 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:49.732180 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "63690cdfb7364ce2b82bcc99fe1e90e1"
format_stamp: "Formatted at 2025-10-27 03:09:49 on dist-test-slave-4cc8"
I20251027 03:09:49.732972 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:49.733378 725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.733516 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.733557 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "63690cdfb7364ce2b82bcc99fe1e90e1"
format_stamp: "Formatted at 2025-10-27 03:09:49 on dist-test-slave-4cc8"
I20251027 03:09:49.733603 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:49.759438 731 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:49.764879 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:49.765070 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:49.765182 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:49.765344 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:49.765370 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.765390 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:49.765419 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:49.768170 731 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:49.768849 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:41207
I20251027 03:09:49.768967 793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:41207 every 8 connection(s)
I20251027 03:09:49.771600 794 heartbeater.cc:344] Connected to a master server at 127.27.151.126:43747
I20251027 03:09:49.771656 794 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:49.771745 794 heartbeater.cc:507] Master 127.27.151.126:43747 requested a full tablet report, sending...
I20251027 03:09:49.771940 387 ts_manager.cc:194] Registered new tserver with Master: 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68:41207)
I20251027 03:09:49.772266 387 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42714
I20251027 03:09:49.777591 693 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:49.799762 702 consensus_queue.cc:1048] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 03:09:49.807505 518 tablet.cc:2378] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 03:09:49.832947 713 consensus_queue.cc:1048] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:09:50.510092 442 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:50.512964 491 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.513096 490 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "84aaafc3085e46c385072b38ef11da96"
mode: GRACEFUL
new_leader_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
from {username='slave'} at 127.0.0.1:32818
I20251027 03:09:50.513212 490 raft_consensus.cc:606] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Received request to transfer leadership to TS a0b7b3f419fb49ac988d86079dbb7c31
I20251027 03:09:50.513578 442 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:50.513628 442 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:50.513612 568 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:50.513604 643 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:50.513815 798 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:09:50.513928 806 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:09:50.514684 643 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
I20251027 03:09:50.514684 568 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
I20251027 03:09:50.515700 387 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 reported cstate change: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "84aaafc3085e46c385072b38ef11da96" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
W20251027 03:09:50.516364 455 consensus_peers.cc:597] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 -> Peer 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68:41207): Couldn't send request to peer 63690cdfb7364ce2b82bcc99fe1e90e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 127bb95bfdee44c9a793316e85dac2fd. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:50.562656 810 raft_consensus.cc:993] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: : Instructing follower a0b7b3f419fb49ac988d86079dbb7c31 to start an election
I20251027 03:09:50.562741 810 raft_consensus.cc:1081] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Signalling peer a0b7b3f419fb49ac988d86079dbb7c31 to start an election
I20251027 03:09:50.562983 568 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
from {username='slave'} at 127.0.0.1:50258
I20251027 03:09:50.563048 568 raft_consensus.cc:493] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:50.563081 568 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:50.563613 568 raft_consensus.cc:515] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.563764 568 leader_election.cc:290] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [CANDIDATE]: Term 2 election: Requested vote from peers 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649), 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553)
I20251027 03:09:50.566243 643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
I20251027 03:09:50.566325 643 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:50.566390 491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "84aaafc3085e46c385072b38ef11da96"
I20251027 03:09:50.566463 491 raft_consensus.cc:3055] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Stepping down as leader of term 1
I20251027 03:09:50.566490 491 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 84aaafc3085e46c385072b38ef11da96, State: Running, Role: LEADER
I20251027 03:09:50.566545 491 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.566634 491 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:09:50.566968 643 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b7b3f419fb49ac988d86079dbb7c31 in term 2.
I20251027 03:09:50.567153 531 leader_election.cc:304] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bbed8d5d5e74d6095dba4ad8c5d8099, a0b7b3f419fb49ac988d86079dbb7c31; no voters:
I20251027 03:09:50.567232 491 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b7b3f419fb49ac988d86079dbb7c31 in term 2.
I20251027 03:09:50.567276 811 raft_consensus.cc:2804] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:09:50.567426 811 raft_consensus.cc:697] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Becoming Leader. State: Replica: a0b7b3f419fb49ac988d86079dbb7c31, State: Running, Role: LEADER
I20251027 03:09:50.567494 811 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.570114 385 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 reported cstate change: term changed from 1 to 2, leader changed from 84aaafc3085e46c385072b38ef11da96 (127.27.151.65) to a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66). New cstate: current_term: 2 leader_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:50.647459 643 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 2 FOLLOWER]: Refusing update from remote peer a0b7b3f419fb49ac988d86079dbb7c31: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:09:50.647709 811 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:09:50.672619 491 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 2 FOLLOWER]: Refusing update from remote peer a0b7b3f419fb49ac988d86079dbb7c31: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:09:50.672837 811 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:09:50.673102 491 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 2 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
I20251027 03:09:50.674331 491 consensus_queue.cc:237] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.674808 568 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:09:50.674849 643 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251027 03:09:50.674921 455 consensus_peers.cc:597] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 -> Peer 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68:41207): Couldn't send request to peer 63690cdfb7364ce2b82bcc99fe1e90e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e270b30eae2f4e079afe00094605a736. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:50.675050 821 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:09:50.675158 806 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:09:50.675971 568 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
I20251027 03:09:50.675971 643 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
I20251027 03:09:50.677065 385 catalog_manager.cc:5649] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 reported cstate change: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "84aaafc3085e46c385072b38ef11da96" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
W20251027 03:09:50.688827 531 consensus_peers.cc:597] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 -> Peer 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68:41207): Couldn't send request to peer 63690cdfb7364ce2b82bcc99fe1e90e1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 127bb95bfdee44c9a793316e85dac2fd. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:09:50.728933 821 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } } }
I20251027 03:09:50.730508 568 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50246
I20251027 03:09:50.730587 568 raft_consensus.cc:606] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Received request to transfer leadership
I20251027 03:09:50.731338 568 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50246
I20251027 03:09:50.731398 568 raft_consensus.cc:606] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Received request to transfer leadership
W20251027 03:09:50.731524 372 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 127bb95bfdee44c9a793316e85dac2fd already in progress [suppressed 1 similar messages]
I20251027 03:09:50.731914 568 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50246
I20251027 03:09:50.731973 568 raft_consensus.cc:606] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Received request to transfer leadership
W20251027 03:09:50.732175 436 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 127bb95bfdee44c9a793316e85dac2fd already in progress
W20251027 03:09:50.732218 436 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 127bb95bfdee44c9a793316e85dac2fd already in progress
I20251027 03:09:50.763713 825 ts_tablet_manager.cc:933] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: Initiating tablet copy from peer 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553)
I20251027 03:09:50.763971 825 tablet_copy_client.cc:323] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:43553
I20251027 03:09:50.767313 501 tablet_copy_service.cc:140] P 84aaafc3085e46c385072b38ef11da96: Received BeginTabletCopySession request for tablet e270b30eae2f4e079afe00094605a736 from peer 63690cdfb7364ce2b82bcc99fe1e90e1 ({username='slave'} at 127.0.0.1:49602)
I20251027 03:09:50.767381 501 tablet_copy_service.cc:161] P 84aaafc3085e46c385072b38ef11da96: Beginning new tablet copy session on tablet e270b30eae2f4e079afe00094605a736 from peer 63690cdfb7364ce2b82bcc99fe1e90e1 at {username='slave'} at 127.0.0.1:49602: session id = 63690cdfb7364ce2b82bcc99fe1e90e1-e270b30eae2f4e079afe00094605a736
I20251027 03:09:50.767786 501 tablet_copy_source_session.cc:215] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:50.768129 825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e270b30eae2f4e079afe00094605a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:50.769096 825 tablet_copy_client.cc:806] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:50.769243 825 tablet_copy_client.cc:670] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:50.769750 825 tablet_copy_client.cc:538] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:50.770295 825 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: Bootstrap starting.
I20251027 03:09:50.771543 825 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:50.771716 825 tablet_bootstrap.cc:492] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: Bootstrap complete.
I20251027 03:09:50.771811 825 ts_tablet_manager.cc:1403] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:09:50.771955 825 raft_consensus.cc:359] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.772032 825 raft_consensus.cc:740] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63690cdfb7364ce2b82bcc99fe1e90e1, State: Initialized, Role: LEARNER
I20251027 03:09:50.772066 825 consensus_queue.cc:260] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.772236 825 ts_tablet_manager.cc:1434] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:09:50.772269 794 heartbeater.cc:499] Master 127.27.151.126:43747 was elected leader, sending a full tablet report...
I20251027 03:09:50.772404 501 tablet_copy_service.cc:342] P 84aaafc3085e46c385072b38ef11da96: Request end of tablet copy session 63690cdfb7364ce2b82bcc99fe1e90e1-e270b30eae2f4e079afe00094605a736 received from {username='slave'} at 127.0.0.1:49602
I20251027 03:09:50.772451 501 tablet_copy_service.cc:434] P 84aaafc3085e46c385072b38ef11da96: ending tablet copy session 63690cdfb7364ce2b82bcc99fe1e90e1-e270b30eae2f4e079afe00094605a736 on tablet e270b30eae2f4e079afe00094605a736 with peer 63690cdfb7364ce2b82bcc99fe1e90e1
I20251027 03:09:50.780522 825 ts_tablet_manager.cc:933] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: Initiating tablet copy from peer a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827)
I20251027 03:09:50.780666 825 tablet_copy_client.cc:323] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:45827
I20251027 03:09:50.782970 578 tablet_copy_service.cc:140] P a0b7b3f419fb49ac988d86079dbb7c31: Received BeginTabletCopySession request for tablet 127bb95bfdee44c9a793316e85dac2fd from peer 63690cdfb7364ce2b82bcc99fe1e90e1 ({username='slave'} at 127.0.0.1:39702)
I20251027 03:09:50.783026 578 tablet_copy_service.cc:161] P a0b7b3f419fb49ac988d86079dbb7c31: Beginning new tablet copy session on tablet 127bb95bfdee44c9a793316e85dac2fd from peer 63690cdfb7364ce2b82bcc99fe1e90e1 at {username='slave'} at 127.0.0.1:39702: session id = 63690cdfb7364ce2b82bcc99fe1e90e1-127bb95bfdee44c9a793316e85dac2fd
I20251027 03:09:50.783396 578 tablet_copy_source_session.cc:215] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:09:50.783699 825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 127bb95bfdee44c9a793316e85dac2fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:50.784530 825 tablet_copy_client.cc:806] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Starting download of 0 data blocks...
I20251027 03:09:50.784677 825 tablet_copy_client.cc:670] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Starting download of 1 WAL segments...
I20251027 03:09:50.785135 825 tablet_copy_client.cc:538] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:09:50.785665 825 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: Bootstrap starting.
I20251027 03:09:50.786957 825 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:09:50.787128 825 tablet_bootstrap.cc:492] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: Bootstrap complete.
I20251027 03:09:50.787221 825 ts_tablet_manager.cc:1403] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:09:50.787407 825 raft_consensus.cc:359] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.787468 825 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 63690cdfb7364ce2b82bcc99fe1e90e1, State: Initialized, Role: LEARNER
I20251027 03:09:50.787488 825 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.787596 825 ts_tablet_manager.cc:1434] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:50.787784 578 tablet_copy_service.cc:342] P a0b7b3f419fb49ac988d86079dbb7c31: Request end of tablet copy session 63690cdfb7364ce2b82bcc99fe1e90e1-127bb95bfdee44c9a793316e85dac2fd received from {username='slave'} at 127.0.0.1:39702
I20251027 03:09:50.787827 578 tablet_copy_service.cc:434] P a0b7b3f419fb49ac988d86079dbb7c31: ending tablet copy session 63690cdfb7364ce2b82bcc99fe1e90e1-127bb95bfdee44c9a793316e85dac2fd on tablet 127bb95bfdee44c9a793316e85dac2fd with peer 63690cdfb7364ce2b82bcc99fe1e90e1
I20251027 03:09:50.845165 811 raft_consensus.cc:993] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: : Instructing follower 9bbed8d5d5e74d6095dba4ad8c5d8099 to start an election
I20251027 03:09:50.845280 811 raft_consensus.cc:1081] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Signalling peer 9bbed8d5d5e74d6095dba4ad8c5d8099 to start an election
I20251027 03:09:50.845511 643 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
from {username='slave'} at 127.0.0.1:36804
I20251027 03:09:50.845618 643 raft_consensus.cc:493] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:50.845659 643 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:50.846542 643 raft_consensus.cc:515] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.846693 643 leader_election.cc:290] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [CANDIDATE]: Term 3 election: Requested vote from peers a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827), 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553)
I20251027 03:09:50.846824 568 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "a0b7b3f419fb49ac988d86079dbb7c31"
I20251027 03:09:50.846824 491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "84aaafc3085e46c385072b38ef11da96"
I20251027 03:09:50.846971 491 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:50.846913 568 raft_consensus.cc:3055] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:09:50.847059 568 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a0b7b3f419fb49ac988d86079dbb7c31, State: Running, Role: LEADER
I20251027 03:09:50.847147 568 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.847223 568 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:09:50.847560 491 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbed8d5d5e74d6095dba4ad8c5d8099 in term 3.
I20251027 03:09:50.847688 568 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bbed8d5d5e74d6095dba4ad8c5d8099 in term 3.
I20251027 03:09:50.847734 607 leader_election.cc:304] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aaafc3085e46c385072b38ef11da96, 9bbed8d5d5e74d6095dba4ad8c5d8099; no voters:
I20251027 03:09:50.847810 702 raft_consensus.cc:2804] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:09:50.847929 702 raft_consensus.cc:697] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Becoming Leader. State: Replica: 9bbed8d5d5e74d6095dba4ad8c5d8099, State: Running, Role: LEADER
I20251027 03:09:50.847990 702 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } }
I20251027 03:09:50.848500 387 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 reported cstate change: term changed from 2 to 3, leader changed from a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66) to 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67). New cstate: current_term: 3 leader_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:09:50.868866 767 raft_consensus.cc:1217] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251027 03:09:50.958495 568 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 3 FOLLOWER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251027 03:09:50.958757 702 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:09:50.963917 490 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251027 03:09:50.964166 702 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:09:50.976224 767 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 2 LEARNER]: Advancing to term 3
I20251027 03:09:50.976980 767 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 LEARNER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20251027 03:09:50.977190 702 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:09:51.002847 798 raft_consensus.cc:1064] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: attempting to promote NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 to VOTER
I20251027 03:09:51.003067 798 consensus_queue.cc:237] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:51.003525 643 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:09:51.003531 767 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 LEARNER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:09:51.003564 568 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:09:51.003793 810 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:09:51.003891 821 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:09:51.004062 798 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:09:51.004810 568 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.004968 767 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.005972 387 catalog_manager.cc:5649] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 reported cstate change: config changed from index 2 to 3, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "84aaafc3085e46c385072b38ef11da96" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.070809 798 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.071844 643 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } attrs { replace: true } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.101001 837 raft_consensus.cc:1064] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: attempting to promote NON_VOTER 63690cdfb7364ce2b82bcc99fe1e90e1 to VOTER
I20251027 03:09:51.101200 837 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:51.101681 767 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 LEARNER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251027 03:09:51.101701 567 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 3 FOLLOWER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251027 03:09:51.101812 491 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20251027 03:09:51.101907 841 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 03:09:51.102031 833 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 03:09:51.102118 840 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 03:09:51.102779 767 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.103842 567 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.105881 491 consensus_queue.cc:237] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:51.106084 384 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 reported cstate change: config changed from index 2 to 5, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.106293 567 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:09:51.106315 767 raft_consensus.cc:1275] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 FOLLOWER]: Refusing update from remote peer 84aaafc3085e46c385072b38ef11da96: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:09:51.106456 806 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:09:51.106551 839 consensus_queue.cc:1048] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:09:51.107255 767 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.107262 567 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.176184 385 catalog_manager.cc:5649] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 reported cstate change: config changed from index 3 to 4, VOTER 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67) evicted. New cstate: current_term: 1 leader_uuid: "84aaafc3085e46c385072b38ef11da96" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.186728 491 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.241161 631 tablet_service.cc:1552] Processing DeleteTablet for tablet e270b30eae2f4e079afe00094605a736 with delete_type TABLET_DATA_TOMBSTONED (TS 9bbed8d5d5e74d6095dba4ad8c5d8099 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50310
I20251027 03:09:51.241413 842 tablet_replica.cc:333] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: stopping tablet replica
I20251027 03:09:51.241492 842 raft_consensus.cc:2243] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:51.241564 842 raft_consensus.cc:2272] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:51.241811 842 ts_tablet_manager.cc:1916] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:51.243624 842 ts_tablet_manager.cc:1929] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251027 03:09:51.243686 842 log.cc:1199] T e270b30eae2f4e079afe00094605a736 P 9bbed8d5d5e74d6095dba4ad8c5d8099: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/e270b30eae2f4e079afe00094605a736
I20251027 03:09:51.243978 372 catalog_manager.cc:4985] TS 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649): tablet e270b30eae2f4e079afe00094605a736 (table test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]) successfully deleted
I20251027 03:09:51.267308 702 raft_consensus.cc:1027] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: attempt to promote peer 63690cdfb7364ce2b82bcc99fe1e90e1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20251027 03:09:51.343173 798 raft_consensus.cc:2955] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.344159 372 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet e270b30eae2f4e079afe00094605a736 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:51.365413 833 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "a0b7b3f419fb49ac988d86079dbb7c31" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 45827 } attrs { replace: true } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.366827 643 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:51.367228 767 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 FOLLOWER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251027 03:09:51.367239 491 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Refusing update from remote peer 9bbed8d5d5e74d6095dba4ad8c5d8099: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20251027 03:09:51.367465 840 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251027 03:09:51.367540 837 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251027 03:09:51.368270 767 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.368320 491 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.369154 384 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 reported cstate change: config changed from index 5 to 6, VOTER a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66) evicted. New cstate: current_term: 3 leader_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.390993 557 tablet_service.cc:1552] Processing DeleteTablet for tablet 127bb95bfdee44c9a793316e85dac2fd with delete_type TABLET_DATA_TOMBSTONED (TS a0b7b3f419fb49ac988d86079dbb7c31 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50246
I20251027 03:09:51.391201 843 tablet_replica.cc:333] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: stopping tablet replica
I20251027 03:09:51.391250 843 raft_consensus.cc:2243] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:51.391295 843 raft_consensus.cc:2272] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:51.391517 843 ts_tablet_manager.cc:1916] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:09:51.392792 843 ts_tablet_manager.cc:1929] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20251027 03:09:51.392841 843 log.cc:1199] T 127bb95bfdee44c9a793316e85dac2fd P a0b7b3f419fb49ac988d86079dbb7c31: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/127bb95bfdee44c9a793316e85dac2fd
I20251027 03:09:51.393139 372 catalog_manager.cc:4985] TS a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66:45827): tablet 127bb95bfdee44c9a793316e85dac2fd (table test-workload [id=4adc8dfa4f964efc9172f84b65f8e3b6]) successfully deleted
I20251027 03:09:51.544909 702 raft_consensus.cc:2955] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER a0b7b3f419fb49ac988d86079dbb7c31 (127.27.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } } }
I20251027 03:09:51.545863 372 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 127bb95bfdee44c9a793316e85dac2fd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:09:52.513816 442 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:52.516644 643 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
mode: GRACEFUL
new_leader_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1"
from {username='slave'} at 127.0.0.1:36820
I20251027 03:09:52.516726 643 raft_consensus.cc:606] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Received request to transfer leadership to TS 63690cdfb7364ce2b82bcc99fe1e90e1
I20251027 03:09:52.516912 442 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251027 03:09:52.516966 442 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:52.615859 702 raft_consensus.cc:993] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: : Instructing follower 63690cdfb7364ce2b82bcc99fe1e90e1 to start an election
I20251027 03:09:52.615952 702 raft_consensus.cc:1081] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Signalling peer 63690cdfb7364ce2b82bcc99fe1e90e1 to start an election
I20251027 03:09:52.616171 767 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd"
dest_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1"
from {username='slave'} at 127.0.0.1:43440
I20251027 03:09:52.616258 767 raft_consensus.cc:493] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 03:09:52.616291 767 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:52.616905 767 raft_consensus.cc:515] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:52.617074 767 leader_election.cc:290] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [CANDIDATE]: Term 4 election: Requested vote from peers 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67:35649), 84aaafc3085e46c385072b38ef11da96 (127.27.151.65:43553)
I20251027 03:09:52.617421 490 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "84aaafc3085e46c385072b38ef11da96"
I20251027 03:09:52.617501 490 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:52.618201 490 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63690cdfb7364ce2b82bcc99fe1e90e1 in term 4.
I20251027 03:09:52.618378 727 leader_election.cc:304] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63690cdfb7364ce2b82bcc99fe1e90e1, 84aaafc3085e46c385072b38ef11da96; no voters:
I20251027 03:09:52.618512 848 raft_consensus.cc:2804] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 4 FOLLOWER]: Leader election won for term 4
I20251027 03:09:52.618671 848 raft_consensus.cc:697] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 4 LEADER]: Becoming Leader. State: Replica: 63690cdfb7364ce2b82bcc99fe1e90e1, State: Running, Role: LEADER
I20251027 03:09:52.618741 848 consensus_queue.cc:237] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:52.619288 386 catalog_manager.cc:5649] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 reported cstate change: term changed from 3 to 4, leader changed from 9bbed8d5d5e74d6095dba4ad8c5d8099 (127.27.151.67) to 63690cdfb7364ce2b82bcc99fe1e90e1 (127.27.151.68). New cstate: current_term: 4 leader_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:09:52.620054 643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "127bb95bfdee44c9a793316e85dac2fd" candidate_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099"
I20251027 03:09:52.620110 643 raft_consensus.cc:3055] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Stepping down as leader of term 3
I20251027 03:09:52.620126 643 raft_consensus.cc:740] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9bbed8d5d5e74d6095dba4ad8c5d8099, State: Running, Role: LEADER
I20251027 03:09:52.620193 643 consensus_queue.cc:260] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 } } peers { permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 } } peers { permanent_uuid: "63690cdfb7364ce2b82bcc99fe1e90e1" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 41207 } attrs { promote: false } }
I20251027 03:09:52.620271 643 raft_consensus.cc:3060] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 3 FOLLOWER]: Advancing to term 4
I20251027 03:09:52.620791 643 raft_consensus.cc:2468] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63690cdfb7364ce2b82bcc99fe1e90e1 in term 4.
I20251027 03:09:52.705384 490 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 4 FOLLOWER]: Refusing update from remote peer 63690cdfb7364ce2b82bcc99fe1e90e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251027 03:09:52.705608 848 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aaafc3085e46c385072b38ef11da96" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251027 03:09:52.744241 643 raft_consensus.cc:1275] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 4 FOLLOWER]: Refusing update from remote peer 63690cdfb7364ce2b82bcc99fe1e90e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20251027 03:09:52.744451 848 consensus_queue.cc:1048] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bbed8d5d5e74d6095dba4ad8c5d8099" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251027 03:09:54.517098 442 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:54.517247 442 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:54.517284 442 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:56.517432 442 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:56.517570 442 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:56.517609 442 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:58.517781 442 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:09:58.517926 442 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:09:58.517959 442 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:09:58.825649 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:09:58.828449 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:58.828550 28253 tablet_replica.cc:333] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96: stopping tablet replica
I20251027 03:09:58.828611 28253 raft_consensus.cc:2243] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:58.828668 28253 raft_consensus.cc:2272] T e270b30eae2f4e079afe00094605a736 P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.828809 28253 tablet_replica.cc:333] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96: stopping tablet replica
I20251027 03:09:58.828835 28253 raft_consensus.cc:2243] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:58.828872 28253 raft_consensus.cc:2272] T 127bb95bfdee44c9a793316e85dac2fd P 84aaafc3085e46c385072b38ef11da96 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.828963 28253 tablet_replica.cc:333] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96: stopping tablet replica
I20251027 03:09:58.828986 28253 raft_consensus.cc:2243] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:58.829010 28253 raft_consensus.cc:2272] T dd03008069ea40cab1a3bec316426350 P 84aaafc3085e46c385072b38ef11da96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.840878 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:09:58.841847 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:09:58.845477 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:58.845575 28253 tablet_replica.cc:333] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31: stopping tablet replica
I20251027 03:09:58.845638 28253 raft_consensus.cc:2243] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:58.845695 28253 raft_consensus.cc:2272] T e270b30eae2f4e079afe00094605a736 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.845805 28253 tablet_replica.cc:333] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31: stopping tablet replica
I20251027 03:09:58.845861 28253 raft_consensus.cc:2243] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:58.845883 28253 raft_consensus.cc:2272] T dd03008069ea40cab1a3bec316426350 P a0b7b3f419fb49ac988d86079dbb7c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.857297 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:09:58.858191 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:09:58.860824 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:58.860931 28253 tablet_replica.cc:333] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099: stopping tablet replica
I20251027 03:09:58.860981 28253 raft_consensus.cc:2243] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:58.861042 28253 raft_consensus.cc:2272] T dd03008069ea40cab1a3bec316426350 P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.861176 28253 tablet_replica.cc:333] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099: stopping tablet replica
I20251027 03:09:58.861205 28253 raft_consensus.cc:2243] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 4 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:58.861235 28253 raft_consensus.cc:2272] T 127bb95bfdee44c9a793316e85dac2fd P 9bbed8d5d5e74d6095dba4ad8c5d8099 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.873220 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:09:58.874120 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:09:58.876837 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:09:58.876955 28253 tablet_replica.cc:333] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1: stopping tablet replica
I20251027 03:09:58.877018 28253 raft_consensus.cc:2243] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 4 LEADER]: Raft consensus shutting down.
I20251027 03:09:58.877074 28253 raft_consensus.cc:2272] T 127bb95bfdee44c9a793316e85dac2fd P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.877182 28253 tablet_replica.cc:333] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1: stopping tablet replica
I20251027 03:09:58.877208 28253 raft_consensus.cc:2243] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:09:58.877239 28253 raft_consensus.cc:2272] T e270b30eae2f4e079afe00094605a736 P 63690cdfb7364ce2b82bcc99fe1e90e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.888563 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:09:58.889292 28253 master.cc:561] Master@127.27.151.126:43747 shutting down...
I20251027 03:09:58.891507 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:09:58.891566 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:09:58.891585 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 50e215f477fb4cf2b0506402920c88ba: stopping tablet replica
I20251027 03:09:58.903107 28253 master.cc:583] Master@127.27.151.126:43747 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10427 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20251027 03:09:58.908412 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:37727
I20251027 03:09:58.908548 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:58.909485 863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.909539 866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.909507 864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:58.909700 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:58.909787 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:58.909816 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:58.909839 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534598909839 us; error 0 us; skew 500 ppm
I20251027 03:09:58.910400 28253 webserver.cc:492] Webserver started at http://127.27.151.126:45375/ using document root <none> and password file <none>
I20251027 03:09:58.910481 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:58.910526 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:58.910569 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:58.910799 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "bd4f92c645a24d30bd0f70955aeab040"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.911527 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:58.911895 871 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.912027 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.912066 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "bd4f92c645a24d30bd0f70955aeab040"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.912110 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:58.918969 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:58.919113 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:58.921332 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:37727
I20251027 03:09:58.924515 923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:37727 every 8 connection(s)
I20251027 03:09:58.924631 924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:09:58.925426 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040: Bootstrap starting.
I20251027 03:09:58.925626 924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040: Neither blocks nor log segments found. Creating new log.
I20251027 03:09:58.926029 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040: No bootstrap required, opened a new log
I20251027 03:09:58.926117 924 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4f92c645a24d30bd0f70955aeab040" member_type: VOTER }
I20251027 03:09:58.926153 924 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:09:58.926165 924 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4f92c645a24d30bd0f70955aeab040, State: Initialized, Role: FOLLOWER
I20251027 03:09:58.926196 924 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4f92c645a24d30bd0f70955aeab040" member_type: VOTER }
I20251027 03:09:58.926239 924 raft_consensus.cc:399] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:09:58.926261 924 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:09:58.926275 924 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:09:58.926699 924 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4f92c645a24d30bd0f70955aeab040" member_type: VOTER }
I20251027 03:09:58.926761 924 leader_election.cc:304] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd4f92c645a24d30bd0f70955aeab040; no voters:
I20251027 03:09:58.926862 924 leader_election.cc:290] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:09:58.926941 927 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:09:58.927000 924 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:09:58.927079 927 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 1 LEADER]: Becoming Leader. State: Replica: bd4f92c645a24d30bd0f70955aeab040, State: Running, Role: LEADER
I20251027 03:09:58.927134 927 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4f92c645a24d30bd0f70955aeab040" member_type: VOTER }
I20251027 03:09:58.927327 928 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd4f92c645a24d30bd0f70955aeab040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4f92c645a24d30bd0f70955aeab040" member_type: VOTER } }
I20251027 03:09:58.927398 928 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:58.927548 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd4f92c645a24d30bd0f70955aeab040. Latest consensus state: current_term: 1 leader_uuid: "bd4f92c645a24d30bd0f70955aeab040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4f92c645a24d30bd0f70955aeab040" member_type: VOTER } }
I20251027 03:09:58.927575 933 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:09:58.927690 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [sys.catalog]: This master's current role is: LEADER
I20251027 03:09:58.927978 933 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:09:58.928467 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:09:58.928563 933 catalog_manager.cc:1357] Generated new cluster ID: 1aa99c6db45c4ddd8126bb1f2e20fb83
I20251027 03:09:58.928604 933 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:09:58.935222 933 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:09:58.935575 933 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:09:58.939821 933 catalog_manager.cc:6022] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040: Generated new TSK 0
I20251027 03:09:58.939899 933 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:09:58.944047 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:58.945057 946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.945156 948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.945210 945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:58.945262 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:58.945351 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:58.945379 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:58.945405 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534598945404 us; error 0 us; skew 500 ppm
I20251027 03:09:58.945904 28253 webserver.cc:492] Webserver started at http://127.27.151.65:46845/ using document root <none> and password file <none>
I20251027 03:09:58.945979 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:58.946023 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:58.946064 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:58.946272 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "77704c9cdbd34233a0dd2fdb047e5dcf"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.947048 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:58.947450 953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.947573 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.947613 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "77704c9cdbd34233a0dd2fdb047e5dcf"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.947657 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:58.960493 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:58.960693 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:58.960815 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:58.960981 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:58.961007 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.961040 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:58.961067 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.963645 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:41103
I20251027 03:09:58.963673 1018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:41103 every 8 connection(s)
I20251027 03:09:58.963941 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:58.964802 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.964923 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:58.964973 28253 server_base.cc:1047] running on GCE node
W20251027 03:09:58.964860 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:58.965157 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:58.965193 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:58.965209 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534598965208 us; error 0 us; skew 500 ppm
I20251027 03:09:58.965765 28253 webserver.cc:492] Webserver started at http://127.27.151.66:46819/ using document root <none> and password file <none>
I20251027 03:09:58.965844 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:58.965898 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:58.965940 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:58.966148 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "765c716ef41a496aa455b790fe68b553"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.967082 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:09:58.967468 1031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.967609 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.967651 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "765c716ef41a496aa455b790fe68b553"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.967697 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:58.971407 1019 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37727
I20251027 03:09:58.971463 1019 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:58.971549 1019 heartbeater.cc:507] Master 127.27.151.126:37727 requested a full tablet report, sending...
I20251027 03:09:58.971771 888 ts_manager.cc:194] Registered new tserver with Master: 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103)
I20251027 03:09:58.972306 888 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60862
I20251027 03:09:58.978581 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:58.978744 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:58.978859 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:58.979022 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:58.979049 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.979076 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:58.979103 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.981602 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:38263
I20251027 03:09:58.981659 1094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:38263 every 8 connection(s)
I20251027 03:09:58.981928 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:09:58.982767 1099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.982786 1098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:09:58.982964 1101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:09:58.983417 28253 server_base.cc:1047] running on GCE node
I20251027 03:09:58.983506 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:09:58.983538 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:09:58.983554 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534598983554 us; error 0 us; skew 500 ppm
I20251027 03:09:58.984056 28253 webserver.cc:492] Webserver started at http://127.27.151.67:44521/ using document root <none> and password file <none>
I20251027 03:09:58.984163 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:09:58.984211 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:09:58.984292 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:09:58.984592 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "ff81bd03990d42d098af1ea7f3b72a1c"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.984901 1095 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37727
I20251027 03:09:58.984952 1095 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:58.985016 1095 heartbeater.cc:507] Master 127.27.151.126:37727 requested a full tablet report, sending...
I20251027 03:09:58.985208 888 ts_manager.cc:194] Registered new tserver with Master: 765c716ef41a496aa455b790fe68b553 (127.27.151.66:38263)
I20251027 03:09:58.985512 888 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60866
I20251027 03:09:58.985594 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:09:58.986022 1107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.986150 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.986189 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "ff81bd03990d42d098af1ea7f3b72a1c"
format_stamp: "Formatted at 2025-10-27 03:09:58 on dist-test-slave-4cc8"
I20251027 03:09:58.986230 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:09:58.994107 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:09:58.994287 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:09:58.994397 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:09:58.994549 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:09:58.994576 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.994603 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:09:58.994629 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:09:58.997591 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:46729
I20251027 03:09:58.997637 1170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:46729 every 8 connection(s)
I20251027 03:09:58.999972 1171 heartbeater.cc:344] Connected to a master server at 127.27.151.126:37727
I20251027 03:09:59.000020 1171 heartbeater.cc:461] Registering TS with master...
I20251027 03:09:59.000084 1171 heartbeater.cc:507] Master 127.27.151.126:37727 requested a full tablet report, sending...
I20251027 03:09:59.000270 888 ts_manager.cc:194] Registered new tserver with Master: ff81bd03990d42d098af1ea7f3b72a1c (127.27.151.67:46729)
I20251027 03:09:59.000540 888 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60874
I20251027 03:09:59.001014 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003270428s
I20251027 03:09:59.972910 1019 heartbeater.cc:499] Master 127.27.151.126:37727 was elected leader, sending a full tablet report...
I20251027 03:09:59.985927 1095 heartbeater.cc:499] Master 127.27.151.126:37727 was elected leader, sending a full tablet report...
I20251027 03:10:00.001020 1171 heartbeater.cc:499] Master 127.27.151.126:37727 was elected leader, sending a full tablet report...
I20251027 03:10:00.025556 28253 test_util.cc:276] Using random seed: 1008211410
I20251027 03:10:00.030172 888 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60884:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:10:00.030288 888 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:10:00.034577 982 tablet_service.cc:1505] Processing CreateTablet for tablet 8c9a2ce014604e56832f149a8c6b583c (DEFAULT_TABLE table=test-workload [id=da7ff2becf9043e49926c4b4f716a1ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:10:00.034709 982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9a2ce014604e56832f149a8c6b583c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:00.035238 1135 tablet_service.cc:1505] Processing CreateTablet for tablet 8c9a2ce014604e56832f149a8c6b583c (DEFAULT_TABLE table=test-workload [id=da7ff2becf9043e49926c4b4f716a1ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:10:00.035348 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9a2ce014604e56832f149a8c6b583c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:00.036134 1191 tablet_bootstrap.cc:492] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf: Bootstrap starting.
I20251027 03:10:00.036329 1059 tablet_service.cc:1505] Processing CreateTablet for tablet 8c9a2ce014604e56832f149a8c6b583c (DEFAULT_TABLE table=test-workload [id=da7ff2becf9043e49926c4b4f716a1ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 03:10:00.036424 1059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9a2ce014604e56832f149a8c6b583c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:00.036535 1191 tablet_bootstrap.cc:654] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:00.036684 1192 tablet_bootstrap.cc:492] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c: Bootstrap starting.
I20251027 03:10:00.037073 1192 tablet_bootstrap.cc:654] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:00.037118 1191 tablet_bootstrap.cc:492] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf: No bootstrap required, opened a new log
I20251027 03:10:00.037181 1191 ts_tablet_manager.cc:1403] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:00.037333 1191 raft_consensus.cc:359] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.037397 1191 raft_consensus.cc:385] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:00.037412 1191 raft_consensus.cc:740] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77704c9cdbd34233a0dd2fdb047e5dcf, State: Initialized, Role: FOLLOWER
I20251027 03:10:00.037454 1191 consensus_queue.cc:260] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.037616 1192 tablet_bootstrap.cc:492] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c: No bootstrap required, opened a new log
I20251027 03:10:00.037670 1192 ts_tablet_manager.cc:1403] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:00.037770 1196 tablet_bootstrap.cc:492] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553: Bootstrap starting.
I20251027 03:10:00.037799 1192 raft_consensus.cc:359] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.037851 1192 raft_consensus.cc:385] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:00.037870 1192 raft_consensus.cc:740] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff81bd03990d42d098af1ea7f3b72a1c, State: Initialized, Role: FOLLOWER
I20251027 03:10:00.037940 1192 consensus_queue.cc:260] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.037613 1191 ts_tablet_manager.cc:1434] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:00.038116 1192 ts_tablet_manager.cc:1434] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:00.038210 1196 tablet_bootstrap.cc:654] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:00.038828 1196 tablet_bootstrap.cc:492] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553: No bootstrap required, opened a new log
I20251027 03:10:00.038879 1196 ts_tablet_manager.cc:1403] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:00.039023 1196 raft_consensus.cc:359] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.039075 1196 raft_consensus.cc:385] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:00.039099 1196 raft_consensus.cc:740] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 765c716ef41a496aa455b790fe68b553, State: Initialized, Role: FOLLOWER
I20251027 03:10:00.039152 1196 consensus_queue.cc:260] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.039294 1196 ts_tablet_manager.cc:1434] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:00.162415 1197 raft_consensus.cc:493] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:00.162520 1197 raft_consensus.cc:515] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.162760 1197 leader_election.cc:290] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103), 765c716ef41a496aa455b790fe68b553 (127.27.151.66:38263)
I20251027 03:10:00.164186 1195 raft_consensus.cc:493] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:00.164255 1195 raft_consensus.cc:515] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.164394 1195 leader_election.cc:290] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 765c716ef41a496aa455b790fe68b553 (127.27.151.66:38263), ff81bd03990d42d098af1ea7f3b72a1c (127.27.151.67:46729)
I20251027 03:10:00.165618 992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c9a2ce014604e56832f149a8c6b583c" candidate_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" is_pre_election: true
I20251027 03:10:00.165699 992 raft_consensus.cc:2468] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff81bd03990d42d098af1ea7f3b72a1c in term 0.
I20251027 03:10:00.165665 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c9a2ce014604e56832f149a8c6b583c" candidate_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765c716ef41a496aa455b790fe68b553" is_pre_election: true
I20251027 03:10:00.165736 1069 raft_consensus.cc:2468] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff81bd03990d42d098af1ea7f3b72a1c in term 0.
I20251027 03:10:00.165846 1109 leader_election.cc:304] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 765c716ef41a496aa455b790fe68b553, ff81bd03990d42d098af1ea7f3b72a1c; no voters:
I20251027 03:10:00.165925 1197 raft_consensus.cc:2804] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:00.165964 1197 raft_consensus.cc:493] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:00.165980 1197 raft_consensus.cc:3060] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:00.166785 1197 raft_consensus.cc:515] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.166908 1197 leader_election.cc:290] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [CANDIDATE]: Term 1 election: Requested vote from peers 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103), 765c716ef41a496aa455b790fe68b553 (127.27.151.66:38263)
I20251027 03:10:00.167019 992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c9a2ce014604e56832f149a8c6b583c" candidate_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf"
I20251027 03:10:00.167066 992 raft_consensus.cc:3060] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:00.167178 1145 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c9a2ce014604e56832f149a8c6b583c" candidate_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" is_pre_election: true
I20251027 03:10:00.167241 1069 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c9a2ce014604e56832f149a8c6b583c" candidate_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765c716ef41a496aa455b790fe68b553"
I20251027 03:10:00.167268 1145 raft_consensus.cc:2393] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77704c9cdbd34233a0dd2fdb047e5dcf in current term 1: Already voted for candidate ff81bd03990d42d098af1ea7f3b72a1c in this term.
I20251027 03:10:00.167284 1069 raft_consensus.cc:3060] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:00.167734 992 raft_consensus.cc:2468] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff81bd03990d42d098af1ea7f3b72a1c in term 1.
I20251027 03:10:00.167851 1110 leader_election.cc:304] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77704c9cdbd34233a0dd2fdb047e5dcf, ff81bd03990d42d098af1ea7f3b72a1c; no voters:
I20251027 03:10:00.167878 1068 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c9a2ce014604e56832f149a8c6b583c" candidate_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "765c716ef41a496aa455b790fe68b553" is_pre_election: true
I20251027 03:10:00.167944 1069 raft_consensus.cc:2468] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff81bd03990d42d098af1ea7f3b72a1c in term 1.
I20251027 03:10:00.168006 1197 raft_consensus.cc:2804] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:00.168061 955 leader_election.cc:304] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77704c9cdbd34233a0dd2fdb047e5dcf; no voters: 765c716ef41a496aa455b790fe68b553, ff81bd03990d42d098af1ea7f3b72a1c
I20251027 03:10:00.168136 1197 raft_consensus.cc:697] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 1 LEADER]: Becoming Leader. State: Replica: ff81bd03990d42d098af1ea7f3b72a1c, State: Running, Role: LEADER
I20251027 03:10:00.168175 1195 raft_consensus.cc:2749] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 03:10:00.168215 1197 consensus_queue.cc:237] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } }
I20251027 03:10:00.168756 888 catalog_manager.cc:5649] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to ff81bd03990d42d098af1ea7f3b72a1c (127.27.151.67). New cstate: current_term: 1 leader_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77704c9cdbd34233a0dd2fdb047e5dcf" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff81bd03990d42d098af1ea7f3b72a1c" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 46729 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:00.175443 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:10:00.177732 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:00.177862 28253 tablet_replica.cc:333] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf: stopping tablet replica
I20251027 03:10:00.177917 28253 raft_consensus.cc:2243] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:00.177946 28253 raft_consensus.cc:2272] T 8c9a2ce014604e56832f149a8c6b583c P 77704c9cdbd34233a0dd2fdb047e5dcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:00.189450 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
W20251027 03:10:00.584753 1110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111)
W20251027 03:10:00.585141 1110 consensus_peers.cc:597] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c -> Peer 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103): Couldn't send request to peer 77704c9cdbd34233a0dd2fdb047e5dcf. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:00.789162 1197 consensus_queue.cc:1048] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "765c716ef41a496aa455b790fe68b553" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 38263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:00.928372 942 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:10:00.928480 942 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:10:00.928498 942 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:10:02.928629 942 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:10:02.928754 942 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:10:02.928774 942 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251027 03:10:03.002281 1110 consensus_peers.cc:597] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c -> Peer 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103): Couldn't send request to peer 77704c9cdbd34233a0dd2fdb047e5dcf. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251027 03:10:04.928903 942 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:10:04.929028 942 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:10:04.929060 942 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251027 03:10:05.509122 1110 consensus_peers.cc:597] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c -> Peer 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103): Couldn't send request to peer 77704c9cdbd34233a0dd2fdb047e5dcf. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20251027 03:10:05.964865 1110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20251027 03:10:06.929200 942 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:10:06.929317 942 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:10:06.929340 942 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251027 03:10:08.104995 1110 consensus_peers.cc:597] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c -> Peer 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103): Couldn't send request to peer 77704c9cdbd34233a0dd2fdb047e5dcf. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20251027 03:10:08.929477 942 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:10:08.929620 942 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:10:08.929646 942 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251027 03:10:10.865819 1110 consensus_peers.cc:597] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c -> Peer 77704c9cdbd34233a0dd2fdb047e5dcf (127.27.151.65:41103): Couldn't send request to peer 77704c9cdbd34233a0dd2fdb047e5dcf. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:41103: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20251027 03:10:10.929797 942 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251027 03:10:10.929951 942 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251027 03:10:10.929983 942 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251027 03:10:11.241106 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:10:11.244186 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:11.244328 28253 tablet_replica.cc:333] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553: stopping tablet replica
I20251027 03:10:11.244395 28253 raft_consensus.cc:2243] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:11.244450 28253 raft_consensus.cc:2272] T 8c9a2ce014604e56832f149a8c6b583c P 765c716ef41a496aa455b790fe68b553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:11.256456 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:10:11.257220 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:10:11.259737 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:11.259827 28253 tablet_replica.cc:333] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c: stopping tablet replica
I20251027 03:10:11.259876 28253 raft_consensus.cc:2243] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:11.259948 28253 raft_consensus.cc:2272] T 8c9a2ce014604e56832f149a8c6b583c P ff81bd03990d42d098af1ea7f3b72a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:11.271926 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:10:11.272639 28253 master.cc:561] Master@127.27.151.126:37727 shutting down...
I20251027 03:10:11.275199 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:11.275256 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:11.275274 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P bd4f92c645a24d30bd0f70955aeab040: stopping tablet replica
I20251027 03:10:11.286564 28253 master.cc:583] Master@127.27.151.126:37727 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12382 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20251027 03:10:11.290863 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:36783
I20251027 03:10:11.290980 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:11.292287 1240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:11.292330 1241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:11.292387 1243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:11.292531 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:11.292627 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:11.292670 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:11.292690 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534611292690 us; error 0 us; skew 500 ppm
I20251027 03:10:11.293202 28253 webserver.cc:492] Webserver started at http://127.27.151.126:46865/ using document root <none> and password file <none>
I20251027 03:10:11.293279 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:11.293318 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:11.293367 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:11.293594 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.294396 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:11.294795 1248 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.294905 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.294952 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.294987 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:11.300869 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:11.300999 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:11.303279 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:36783
I20251027 03:10:11.305181 1300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:36783 every 8 connection(s)
I20251027 03:10:11.305274 1301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:11.306142 1301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157: Bootstrap starting.
I20251027 03:10:11.306355 1301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:11.306784 1301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157: No bootstrap required, opened a new log
I20251027 03:10:11.306879 1301 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" member_type: VOTER }
I20251027 03:10:11.306928 1301 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:11.306941 1301 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7d56cf207e438f8b7fc7ccbfa7f157, State: Initialized, Role: FOLLOWER
I20251027 03:10:11.306981 1301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" member_type: VOTER }
I20251027 03:10:11.307024 1301 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:10:11.307040 1301 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:10:11.307060 1301 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:11.307490 1301 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" member_type: VOTER }
I20251027 03:10:11.307538 1301 leader_election.cc:304] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b7d56cf207e438f8b7fc7ccbfa7f157; no voters:
I20251027 03:10:11.307618 1301 leader_election.cc:290] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:10:11.307672 1304 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:11.307751 1301 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:10:11.307785 1304 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7d56cf207e438f8b7fc7ccbfa7f157, State: Running, Role: LEADER
I20251027 03:10:11.307826 1304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" member_type: VOTER }
I20251027 03:10:11.308007 1305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" member_type: VOTER } }
I20251027 03:10:11.308012 1306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7d56cf207e438f8b7fc7ccbfa7f157. Latest consensus state: current_term: 1 leader_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d56cf207e438f8b7fc7ccbfa7f157" member_type: VOTER } }
I20251027 03:10:11.308071 1305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [sys.catalog]: This master's current role is: LEADER
I20251027 03:10:11.308074 1306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [sys.catalog]: This master's current role is: LEADER
I20251027 03:10:11.308272 1309 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:10:11.308419 1309 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:10:11.309075 1309 catalog_manager.cc:1357] Generated new cluster ID: 740a857d07cc4382aeb6a776dd18e460
I20251027 03:10:11.309130 1309 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:10:11.309132 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:10:11.326575 1309 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:10:11.326994 1309 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:10:11.333869 1309 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157: Generated new TSK 0
I20251027 03:10:11.333946 1309 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:10:11.340801 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:11.341933 1323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:11.341998 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:11.342075 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:11.341953 1325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:11.342207 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:11.342239 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:11.342258 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534611342258 us; error 0 us; skew 500 ppm
I20251027 03:10:11.342756 28253 webserver.cc:492] Webserver started at http://127.27.151.65:36141/ using document root <none> and password file <none>
I20251027 03:10:11.342840 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:11.342895 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:11.342942 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:11.343159 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "ba1f79e4d8aa4f7790466a6eafb6f065"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.343969 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:11.344400 1330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.344512 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.344551 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "ba1f79e4d8aa4f7790466a6eafb6f065"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.344594 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:11.360496 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:11.360668 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:11.360787 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:11.360956 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:11.360981 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.361011 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:11.361040 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.363502 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:45329
I20251027 03:10:11.363528 1394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:45329 every 8 connection(s)
I20251027 03:10:11.363803 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:11.364737 1399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:11.364835 1402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:11.364892 1400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:11.365214 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:11.365314 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:11.365340 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:11.365360 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534611365360 us; error 0 us; skew 500 ppm
I20251027 03:10:11.365918 28253 webserver.cc:492] Webserver started at http://127.27.151.66:33643/ using document root <none> and password file <none>
I20251027 03:10:11.366000 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:11.366040 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:11.366086 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:11.366380 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "7771ba8de9194e7f9242c19e43d9064a"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.366983 1395 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36783
I20251027 03:10:11.367033 1395 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:11.367100 1395 heartbeater.cc:507] Master 127.27.151.126:36783 requested a full tablet report, sending...
I20251027 03:10:11.367281 1265 ts_manager.cc:194] Registered new tserver with Master: ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329)
I20251027 03:10:11.367301 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:11.367704 1407 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.367791 1265 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50512
I20251027 03:10:11.367808 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.367837 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "7771ba8de9194e7f9242c19e43d9064a"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.367871 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:11.375422 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:11.375556 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:11.375640 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:11.375789 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:11.375815 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.375845 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:11.375872 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.378448 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:34825
I20251027 03:10:11.378497 1470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:34825 every 8 connection(s)
I20251027 03:10:11.378772 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:11.379738 1474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:11.379845 1475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:11.379971 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:11.380059 1477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:11.380208 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:11.380241 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:11.380259 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534611380258 us; error 0 us; skew 500 ppm
I20251027 03:10:11.380798 28253 webserver.cc:492] Webserver started at http://127.27.151.67:45381/ using document root <none> and password file <none>
I20251027 03:10:11.380883 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:11.380931 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:11.380981 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:11.381235 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.381736 1471 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36783
I20251027 03:10:11.381795 1471 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:11.381907 1471 heartbeater.cc:507] Master 127.27.151.126:36783 requested a full tablet report, sending...
I20251027 03:10:11.382110 1265 ts_manager.cc:194] Registered new tserver with Master: 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:11.382248 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:11.382467 1265 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50524
I20251027 03:10:11.382752 1482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.382865 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.382915 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9"
format_stamp: "Formatted at 2025-10-27 03:10:11 on dist-test-slave-4cc8"
I20251027 03:10:11.382954 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:11.388814 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:11.389020 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:11.389148 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:11.389302 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:11.389341 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.389360 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:11.389387 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:11.392385 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:41745
I20251027 03:10:11.392419 1545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:41745 every 8 connection(s)
I20251027 03:10:11.394748 1546 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36783
I20251027 03:10:11.394797 1546 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:11.394879 1546 heartbeater.cc:507] Master 127.27.151.126:36783 requested a full tablet report, sending...
I20251027 03:10:11.395051 1265 ts_manager.cc:194] Registered new tserver with Master: 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:11.395352 1265 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50536
I20251027 03:10:11.395789 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003259821s
I20251027 03:10:12.368211 1395 heartbeater.cc:499] Master 127.27.151.126:36783 was elected leader, sending a full tablet report...
I20251027 03:10:12.382921 1471 heartbeater.cc:499] Master 127.27.151.126:36783 was elected leader, sending a full tablet report...
I20251027 03:10:12.395759 1546 heartbeater.cc:499] Master 127.27.151.126:36783 was elected leader, sending a full tablet report...
I20251027 03:10:12.420518 28253 test_util.cc:276] Using random seed: 1020606372
I20251027 03:10:12.425210 1265 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50540:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:10:12.425359 1265 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:10:12.430043 1435 tablet_service.cc:1505] Processing CreateTablet for tablet 91ce3a308ed84dac97f990e5b549788c (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:12.430157 1358 tablet_service.cc:1505] Processing CreateTablet for tablet 91ce3a308ed84dac97f990e5b549788c (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:12.430254 1434 tablet_service.cc:1505] Processing CreateTablet for tablet 1e74f97f28b04ffd91c671ac78d0f66d (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:12.430282 1358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce3a308ed84dac97f990e5b549788c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.430357 1433 tablet_service.cc:1505] Processing CreateTablet for tablet 85d903e62b144ce3aff41aed79cb38a1 (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:12.430485 1432 tablet_service.cc:1505] Processing CreateTablet for tablet 98fcd7f5723d43448aa21d2095170c98 (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:12.430603 1357 tablet_service.cc:1505] Processing CreateTablet for tablet 1e74f97f28b04ffd91c671ac78d0f66d (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:12.430692 1357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e74f97f28b04ffd91c671ac78d0f66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.430845 1510 tablet_service.cc:1505] Processing CreateTablet for tablet 91ce3a308ed84dac97f990e5b549788c (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:12.430877 1509 tablet_service.cc:1505] Processing CreateTablet for tablet 1e74f97f28b04ffd91c671ac78d0f66d (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:12.430919 1510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce3a308ed84dac97f990e5b549788c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.430982 1509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e74f97f28b04ffd91c671ac78d0f66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.432016 1356 tablet_service.cc:1505] Processing CreateTablet for tablet 85d903e62b144ce3aff41aed79cb38a1 (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:12.432114 1356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d903e62b144ce3aff41aed79cb38a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.432315 1508 tablet_service.cc:1505] Processing CreateTablet for tablet 85d903e62b144ce3aff41aed79cb38a1 (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:12.432415 1508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d903e62b144ce3aff41aed79cb38a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.432525 1567 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9: Bootstrap starting.
I20251027 03:10:12.432983 1567 tablet_bootstrap.cc:654] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.433395 1507 tablet_service.cc:1505] Processing CreateTablet for tablet 98fcd7f5723d43448aa21d2095170c98 (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:12.433477 1507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fcd7f5723d43448aa21d2095170c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.433493 1355 tablet_service.cc:1505] Processing CreateTablet for tablet 98fcd7f5723d43448aa21d2095170c98 (DEFAULT_TABLE table=test-workload [id=0e6f94da7f6d4c8cad3271d915238216]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:12.433580 1355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fcd7f5723d43448aa21d2095170c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.433696 1567 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9: No bootstrap required, opened a new log
I20251027 03:10:12.433728 1567 ts_tablet_manager.cc:1403] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.433857 1567 raft_consensus.cc:359] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.433905 1567 raft_consensus.cc:385] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.433924 1567 raft_consensus.cc:740] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0eeca3976f4cd391a5b74c7aa6cee9, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.433990 1567 consensus_queue.cc:260] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.434137 1567 ts_tablet_manager.cc:1434] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:12.434188 1567 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9: Bootstrap starting.
I20251027 03:10:12.434684 1567 tablet_bootstrap.cc:654] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.430182 1435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce3a308ed84dac97f990e5b549788c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.434913 1433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d903e62b144ce3aff41aed79cb38a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.435392 1567 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9: No bootstrap required, opened a new log
I20251027 03:10:12.435443 1567 ts_tablet_manager.cc:1403] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.435581 1567 raft_consensus.cc:359] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.435657 1567 raft_consensus.cc:385] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.435678 1567 raft_consensus.cc:740] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0eeca3976f4cd391a5b74c7aa6cee9, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.435717 1567 consensus_queue.cc:260] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.435792 1567 ts_tablet_manager.cc:1434] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.435835 1567 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Bootstrap starting.
I20251027 03:10:12.436128 1571 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a: Bootstrap starting.
I20251027 03:10:12.436225 1567 tablet_bootstrap.cc:654] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.436230 1566 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: Bootstrap starting.
I20251027 03:10:12.436524 1432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fcd7f5723d43448aa21d2095170c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.436581 1571 tablet_bootstrap.cc:654] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.436707 1566 tablet_bootstrap.cc:654] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.436980 1567 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: No bootstrap required, opened a new log
I20251027 03:10:12.437031 1567 ts_tablet_manager.cc:1403] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:12.437184 1567 raft_consensus.cc:359] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.437249 1567 raft_consensus.cc:385] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.437274 1567 raft_consensus.cc:740] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0eeca3976f4cd391a5b74c7aa6cee9, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.437321 1567 consensus_queue.cc:260] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.437422 1567 ts_tablet_manager.cc:1434] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.437436 1571 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a: No bootstrap required, opened a new log
I20251027 03:10:12.437326 1566 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: No bootstrap required, opened a new log
I20251027 03:10:12.437477 1567 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Bootstrap starting.
I20251027 03:10:12.437520 1566 ts_tablet_manager.cc:1403] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.437469 1571 ts_tablet_manager.cc:1403] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:12.437666 1566 raft_consensus.cc:359] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.437731 1566 raft_consensus.cc:385] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.437697 1571 raft_consensus.cc:359] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.437669 1434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e74f97f28b04ffd91c671ac78d0f66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:12.437783 1571 raft_consensus.cc:385] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.437829 1571 raft_consensus.cc:740] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7771ba8de9194e7f9242c19e43d9064a, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.437877 1571 consensus_queue.cc:260] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.437883 1567 tablet_bootstrap.cc:654] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.438033 1571 ts_tablet_manager.cc:1434] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.438107 1571 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: Bootstrap starting.
I20251027 03:10:12.438494 1571 tablet_bootstrap.cc:654] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.438553 1567 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: No bootstrap required, opened a new log
I20251027 03:10:12.438606 1567 ts_tablet_manager.cc:1403] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:12.438732 1567 raft_consensus.cc:359] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.438777 1567 raft_consensus.cc:385] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.438800 1567 raft_consensus.cc:740] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f0eeca3976f4cd391a5b74c7aa6cee9, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.438848 1567 consensus_queue.cc:260] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.438925 1567 ts_tablet_manager.cc:1434] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.437769 1566 raft_consensus.cc:740] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1f79e4d8aa4f7790466a6eafb6f065, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.439218 1566 consensus_queue.cc:260] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.439365 1566 ts_tablet_manager.cc:1434] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 03:10:12.439436 1566 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065: Bootstrap starting.
I20251027 03:10:12.439518 1571 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: No bootstrap required, opened a new log
I20251027 03:10:12.439554 1571 ts_tablet_manager.cc:1403] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:12.439709 1571 raft_consensus.cc:359] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.439759 1571 raft_consensus.cc:385] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.439782 1571 raft_consensus.cc:740] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7771ba8de9194e7f9242c19e43d9064a, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.439833 1571 consensus_queue.cc:260] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.439877 1566 tablet_bootstrap.cc:654] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.439922 1571 ts_tablet_manager.cc:1434] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.439985 1571 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a: Bootstrap starting.
I20251027 03:10:12.440393 1571 tablet_bootstrap.cc:654] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.440500 1566 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065: No bootstrap required, opened a new log
I20251027 03:10:12.440539 1566 ts_tablet_manager.cc:1403] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.440706 1566 raft_consensus.cc:359] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.440766 1566 raft_consensus.cc:385] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.440788 1566 raft_consensus.cc:740] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1f79e4d8aa4f7790466a6eafb6f065, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.440831 1566 consensus_queue.cc:260] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.440924 1566 ts_tablet_manager.cc:1434] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.440984 1566 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065: Bootstrap starting.
I20251027 03:10:12.440990 1571 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a: No bootstrap required, opened a new log
I20251027 03:10:12.441040 1571 ts_tablet_manager.cc:1403] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.441176 1571 raft_consensus.cc:359] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.441243 1571 raft_consensus.cc:385] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.441272 1571 raft_consensus.cc:740] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7771ba8de9194e7f9242c19e43d9064a, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.441319 1571 consensus_queue.cc:260] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.441382 1566 tablet_bootstrap.cc:654] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.441457 1571 ts_tablet_manager.cc:1434] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.441521 1571 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: Bootstrap starting.
I20251027 03:10:12.441931 1571 tablet_bootstrap.cc:654] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.442083 1566 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065: No bootstrap required, opened a new log
I20251027 03:10:12.442130 1566 ts_tablet_manager.cc:1403] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.442358 1566 raft_consensus.cc:359] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.442410 1566 raft_consensus.cc:385] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.442426 1566 raft_consensus.cc:740] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1f79e4d8aa4f7790466a6eafb6f065, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.442487 1566 consensus_queue.cc:260] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.442566 1571 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: No bootstrap required, opened a new log
I20251027 03:10:12.442600 1566 ts_tablet_manager.cc:1434] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.442660 1571 ts_tablet_manager.cc:1403] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.442682 1566 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065: Bootstrap starting.
I20251027 03:10:12.442828 1571 raft_consensus.cc:359] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.442950 1571 raft_consensus.cc:385] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.443024 1571 raft_consensus.cc:740] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7771ba8de9194e7f9242c19e43d9064a, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.443110 1571 consensus_queue.cc:260] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.443291 1571 ts_tablet_manager.cc:1434] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:12.443382 1566 tablet_bootstrap.cc:654] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:12.443883 1566 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065: No bootstrap required, opened a new log
I20251027 03:10:12.443929 1566 ts_tablet_manager.cc:1403] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:12.444204 1566 raft_consensus.cc:359] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.444281 1566 raft_consensus.cc:385] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:12.444305 1566 raft_consensus.cc:740] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1f79e4d8aa4f7790466a6eafb6f065, State: Initialized, Role: FOLLOWER
I20251027 03:10:12.444362 1566 consensus_queue.cc:260] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.444446 1566 ts_tablet_manager.cc:1434] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:12.460219 1575 raft_consensus.cc:493] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:12.460270 1575 raft_consensus.cc:515] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.460426 1575 leader_election.cc:290] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:12.463044 1520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e74f97f28b04ffd91c671ac78d0f66d" candidate_uuid: "7771ba8de9194e7f9242c19e43d9064a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" is_pre_election: true
I20251027 03:10:12.463106 1520 raft_consensus.cc:2468] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7771ba8de9194e7f9242c19e43d9064a in term 0.
I20251027 03:10:12.463148 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e74f97f28b04ffd91c671ac78d0f66d" candidate_uuid: "7771ba8de9194e7f9242c19e43d9064a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" is_pre_election: true
I20251027 03:10:12.463222 1368 raft_consensus.cc:2468] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7771ba8de9194e7f9242c19e43d9064a in term 0.
I20251027 03:10:12.463238 1410 leader_election.cc:304] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, 7771ba8de9194e7f9242c19e43d9064a; no voters:
I20251027 03:10:12.463354 1575 raft_consensus.cc:2804] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:12.463402 1575 raft_consensus.cc:493] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:12.463421 1575 raft_consensus.cc:3060] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.463929 1575 raft_consensus.cc:515] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.464033 1575 leader_election.cc:290] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 1 election: Requested vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:12.464172 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e74f97f28b04ffd91c671ac78d0f66d" candidate_uuid: "7771ba8de9194e7f9242c19e43d9064a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065"
I20251027 03:10:12.464179 1520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e74f97f28b04ffd91c671ac78d0f66d" candidate_uuid: "7771ba8de9194e7f9242c19e43d9064a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9"
I20251027 03:10:12.464236 1368 raft_consensus.cc:3060] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.464244 1520 raft_consensus.cc:3060] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.464797 1368 raft_consensus.cc:2468] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7771ba8de9194e7f9242c19e43d9064a in term 1.
I20251027 03:10:12.464823 1520 raft_consensus.cc:2468] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7771ba8de9194e7f9242c19e43d9064a in term 1.
I20251027 03:10:12.464955 1410 leader_election.cc:304] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, 7771ba8de9194e7f9242c19e43d9064a, ba1f79e4d8aa4f7790466a6eafb6f065; no voters:
I20251027 03:10:12.465039 1575 raft_consensus.cc:2804] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:12.465162 1575 raft_consensus.cc:697] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 LEADER]: Becoming Leader. State: Replica: 7771ba8de9194e7f9242c19e43d9064a, State: Running, Role: LEADER
I20251027 03:10:12.465206 1575 consensus_queue.cc:237] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.465680 1265 catalog_manager.cc:5649] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66). New cstate: current_term: 1 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:12.470990 1569 raft_consensus.cc:493] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:12.471043 1569 raft_consensus.cc:515] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.471140 1569 leader_election.cc:290] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:12.473443 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fcd7f5723d43448aa21d2095170c98" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" is_pre_election: true
I20251027 03:10:12.473455 1445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fcd7f5723d43448aa21d2095170c98" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7771ba8de9194e7f9242c19e43d9064a" is_pre_election: true
I20251027 03:10:12.473528 1368 raft_consensus.cc:2468] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 0.
I20251027 03:10:12.473533 1445 raft_consensus.cc:2468] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 0.
I20251027 03:10:12.473711 1484 leader_election.cc:304] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, ba1f79e4d8aa4f7790466a6eafb6f065; no voters:
I20251027 03:10:12.473796 1569 raft_consensus.cc:2804] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:12.473834 1569 raft_consensus.cc:493] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:12.473850 1569 raft_consensus.cc:3060] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.474256 1569 raft_consensus.cc:515] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.474339 1569 leader_election.cc:290] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 election: Requested vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:12.474490 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fcd7f5723d43448aa21d2095170c98" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065"
I20251027 03:10:12.474519 1445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98fcd7f5723d43448aa21d2095170c98" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7771ba8de9194e7f9242c19e43d9064a"
I20251027 03:10:12.474550 1368 raft_consensus.cc:3060] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.474555 1445 raft_consensus.cc:3060] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.475029 1368 raft_consensus.cc:2468] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 1.
I20251027 03:10:12.475137 1445 raft_consensus.cc:2468] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 1.
I20251027 03:10:12.475178 1484 leader_election.cc:304] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, ba1f79e4d8aa4f7790466a6eafb6f065; no voters:
I20251027 03:10:12.475260 1569 raft_consensus.cc:2804] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:12.475376 1569 raft_consensus.cc:697] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0eeca3976f4cd391a5b74c7aa6cee9, State: Running, Role: LEADER
I20251027 03:10:12.475433 1569 consensus_queue.cc:237] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.475867 1265 catalog_manager.cc:5649] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:12.624958 1569 raft_consensus.cc:493] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:12.625052 1569 raft_consensus.cc:515] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.625175 1569 leader_election.cc:290] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:12.625356 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85d903e62b144ce3aff41aed79cb38a1" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" is_pre_election: true
I20251027 03:10:12.625371 1445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85d903e62b144ce3aff41aed79cb38a1" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7771ba8de9194e7f9242c19e43d9064a" is_pre_election: true
I20251027 03:10:12.625442 1368 raft_consensus.cc:2468] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 0.
I20251027 03:10:12.625456 1445 raft_consensus.cc:2468] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 0.
I20251027 03:10:12.625568 1484 leader_election.cc:304] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, ba1f79e4d8aa4f7790466a6eafb6f065; no voters:
I20251027 03:10:12.625655 1569 raft_consensus.cc:2804] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:12.625698 1569 raft_consensus.cc:493] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:12.625731 1569 raft_consensus.cc:3060] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.626341 1569 raft_consensus.cc:515] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.626448 1569 leader_election.cc:290] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 election: Requested vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:12.626587 1368 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85d903e62b144ce3aff41aed79cb38a1" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065"
I20251027 03:10:12.626609 1445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85d903e62b144ce3aff41aed79cb38a1" candidate_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7771ba8de9194e7f9242c19e43d9064a"
I20251027 03:10:12.626652 1368 raft_consensus.cc:3060] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.626665 1445 raft_consensus.cc:3060] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.627180 1445 raft_consensus.cc:2468] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 1.
I20251027 03:10:12.627187 1368 raft_consensus.cc:2468] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f0eeca3976f4cd391a5b74c7aa6cee9 in term 1.
I20251027 03:10:12.627337 1484 leader_election.cc:304] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, 7771ba8de9194e7f9242c19e43d9064a; no voters:
I20251027 03:10:12.627413 1569 raft_consensus.cc:2804] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:12.627465 1569 raft_consensus.cc:697] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f0eeca3976f4cd391a5b74c7aa6cee9, State: Running, Role: LEADER
I20251027 03:10:12.627514 1569 consensus_queue.cc:237] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } }
I20251027 03:10:12.628065 1265 catalog_manager.cc:5649] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67). New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:12.819829 1578 raft_consensus.cc:493] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:12.819926 1578 raft_consensus.cc:515] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.820083 1578 leader_election.cc:290] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745), 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:12.822613 1520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91ce3a308ed84dac97f990e5b549788c" candidate_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" is_pre_election: true
I20251027 03:10:12.822621 1445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91ce3a308ed84dac97f990e5b549788c" candidate_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7771ba8de9194e7f9242c19e43d9064a" is_pre_election: true
I20251027 03:10:12.822701 1520 raft_consensus.cc:2468] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1f79e4d8aa4f7790466a6eafb6f065 in term 0.
I20251027 03:10:12.822746 1445 raft_consensus.cc:2468] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1f79e4d8aa4f7790466a6eafb6f065 in term 0.
I20251027 03:10:12.822885 1333 leader_election.cc:304] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, ba1f79e4d8aa4f7790466a6eafb6f065; no voters:
I20251027 03:10:12.822985 1578 raft_consensus.cc:2804] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:12.823030 1578 raft_consensus.cc:493] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:12.823058 1578 raft_consensus.cc:3060] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.823657 1578 raft_consensus.cc:515] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.823773 1578 leader_election.cc:290] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [CANDIDATE]: Term 1 election: Requested vote from peers 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745), 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:12.823890 1520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91ce3a308ed84dac97f990e5b549788c" candidate_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9"
I20251027 03:10:12.823906 1445 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91ce3a308ed84dac97f990e5b549788c" candidate_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7771ba8de9194e7f9242c19e43d9064a"
I20251027 03:10:12.823941 1520 raft_consensus.cc:3060] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.823956 1445 raft_consensus.cc:3060] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:12.824484 1445 raft_consensus.cc:2468] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba1f79e4d8aa4f7790466a6eafb6f065 in term 1.
I20251027 03:10:12.824499 1520 raft_consensus.cc:2468] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba1f79e4d8aa4f7790466a6eafb6f065 in term 1.
I20251027 03:10:12.824616 1333 leader_election.cc:304] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, ba1f79e4d8aa4f7790466a6eafb6f065; no voters:
I20251027 03:10:12.824687 1578 raft_consensus.cc:2804] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:12.824803 1578 raft_consensus.cc:697] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 LEADER]: Becoming Leader. State: Replica: ba1f79e4d8aa4f7790466a6eafb6f065, State: Running, Role: LEADER
I20251027 03:10:12.824855 1578 consensus_queue.cc:237] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:12.825337 1265 catalog_manager.cc:5649] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:12.864490 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:10:12.866855 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:12.867027 28253 tablet_replica.cc:333] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065: stopping tablet replica
I20251027 03:10:12.867079 28253 raft_consensus.cc:2243] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:12.867123 28253 raft_consensus.cc:2272] T 98fcd7f5723d43448aa21d2095170c98 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:12.867288 28253 tablet_replica.cc:333] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: stopping tablet replica
I20251027 03:10:12.867327 28253 raft_consensus.cc:2243] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:12.867374 28253 pending_rounds.cc:70] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: Trying to abort 1 pending ops.
I20251027 03:10:12.867398 28253 pending_rounds.cc:77] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245774131208192 op_type: NO_OP noop_request { }
I20251027 03:10:12.867446 28253 raft_consensus.cc:2889] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:12.867477 28253 raft_consensus.cc:2272] T 91ce3a308ed84dac97f990e5b549788c P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:12.867705 28253 tablet_replica.cc:333] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065: stopping tablet replica
I20251027 03:10:12.867754 28253 raft_consensus.cc:2243] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:12.867794 28253 raft_consensus.cc:2272] T 85d903e62b144ce3aff41aed79cb38a1 P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:12.867949 28253 tablet_replica.cc:333] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065: stopping tablet replica
I20251027 03:10:12.867983 28253 raft_consensus.cc:2243] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:12.868019 28253 raft_consensus.cc:2272] T 1e74f97f28b04ffd91c671ac78d0f66d P ba1f79e4d8aa4f7790466a6eafb6f065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:12.869714 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
W20251027 03:10:12.894332 1484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251027 03:10:12.894652 1484 consensus_peers.cc:597] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:12.964623 1569 consensus_queue.cc:1048] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:12.995548 1575 consensus_queue.cc:1048] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 03:10:12.997241 1409 consensus_peers.cc:597] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:13.175910 1590 consensus_queue.cc:1048] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 03:10:13.177471 1484 consensus_peers.cc:597] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:13.522727 1575 consensus_queue.cc:579] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.057s)
W20251027 03:10:13.523384 1265 catalog_manager.cc:5327] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1e74f97f28b04ffd91c671ac78d0f66d with cas_config_opid_index -1: no extra replica candidate found for tablet 1e74f97f28b04ffd91c671ac78d0f66d (table test-workload [id=0e6f94da7f6d4c8cad3271d915238216]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20251027 03:10:13.773204 1590 consensus_queue.cc:579] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.146s)
I20251027 03:10:13.817370 1609 consensus_queue.cc:579] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.342s)
I20251027 03:10:13.870818 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:13.871935 1614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:13.871948 1611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:13.872078 1612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:13.872174 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:13.872252 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:13.872283 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:13.872305 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534613872305 us; error 0 us; skew 500 ppm
I20251027 03:10:13.872890 28253 webserver.cc:492] Webserver started at http://127.27.151.68:44091/ using document root <none> and password file <none>
I20251027 03:10:13.872972 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:13.873023 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:13.873054 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:13.873283 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "337d89be988d4793bc3f024a5e32096a"
format_stamp: "Formatted at 2025-10-27 03:10:13 on dist-test-slave-4cc8"
I20251027 03:10:13.874141 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:13.874538 1619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:13.874680 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:13.874727 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "337d89be988d4793bc3f024a5e32096a"
format_stamp: "Formatted at 2025-10-27 03:10:13 on dist-test-slave-4cc8"
I20251027 03:10:13.874770 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:13.880627 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:13.880801 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:13.880923 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:13.881088 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:13.881114 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:13.881142 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:13.881170 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:13.883649 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:44013
I20251027 03:10:13.884172 1682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:44013 every 8 connection(s)
I20251027 03:10:13.886474 1683 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36783
I20251027 03:10:13.886529 1683 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:13.886621 1683 heartbeater.cc:507] Master 127.27.151.126:36783 requested a full tablet report, sending...
I20251027 03:10:13.886817 1265 ts_manager.cc:194] Registered new tserver with Master: 337d89be988d4793bc3f024a5e32096a (127.27.151.68:44013)
I20251027 03:10:13.887125 1265 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50552
I20251027 03:10:13.903942 1520 consensus_queue.cc:237] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:13.904537 1445 raft_consensus.cc:1275] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:13.904732 1609 consensus_queue.cc:1048] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251027 03:10:13.904917 1484 consensus_peers.cc:597] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:13.905437 1590 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:13.905545 1445 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:13.906378 1251 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 98fcd7f5723d43448aa21d2095170c98 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20251027 03:10:13.906538 1265 catalog_manager.cc:5649] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a reported cstate change: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
W20251027 03:10:13.907271 1485 consensus_peers.cc:597] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer 337d89be988d4793bc3f024a5e32096a (127.27.151.68:44013): Couldn't send request to peer 337d89be988d4793bc3f024a5e32096a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98fcd7f5723d43448aa21d2095170c98. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:13.948122 1520 consensus_queue.cc:237] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
W20251027 03:10:13.948720 1485 consensus_peers.cc:597] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer 337d89be988d4793bc3f024a5e32096a (127.27.151.68:44013): Couldn't send request to peer 337d89be988d4793bc3f024a5e32096a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85d903e62b144ce3aff41aed79cb38a1. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:13.948755 1445 raft_consensus.cc:1275] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:13.949009 1590 consensus_queue.cc:1048] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251027 03:10:13.949084 1484 consensus_peers.cc:597] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:13.949616 1609 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:13.949764 1445 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:13.950634 1251 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 85d903e62b144ce3aff41aed79cb38a1 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20251027 03:10:13.950769 1264 catalog_manager.cc:5649] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a reported cstate change: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:14.121651 1445 consensus_queue.cc:237] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.122430 1520 raft_consensus.cc:1275] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:14.122629 1698 consensus_queue.cc:1048] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251027 03:10:14.123153 1409 consensus_peers.cc:597] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:14.123441 1697 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:14.123596 1520 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:14.124727 1265 catalog_manager.cc:5649] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:14.125070 1250 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1e74f97f28b04ffd91c671ac78d0f66d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
W20251027 03:10:14.125360 1410 consensus_peers.cc:597] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a -> Peer 337d89be988d4793bc3f024a5e32096a (127.27.151.68:44013): Couldn't send request to peer 337d89be988d4793bc3f024a5e32096a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e74f97f28b04ffd91c671ac78d0f66d. This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:14.309172 1314 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver ba1f79e4d8aa4f7790466a6eafb6f065 not available for placement
I20251027 03:10:14.330401 1701 ts_tablet_manager.cc:933] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: Initiating tablet copy from peer 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:14.330582 1701 tablet_copy_client.cc:323] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:41745
I20251027 03:10:14.332911 1530 tablet_copy_service.cc:140] P 0f0eeca3976f4cd391a5b74c7aa6cee9: Received BeginTabletCopySession request for tablet 98fcd7f5723d43448aa21d2095170c98 from peer 337d89be988d4793bc3f024a5e32096a ({username='slave'} at 127.0.0.1:53628)
I20251027 03:10:14.332962 1530 tablet_copy_service.cc:161] P 0f0eeca3976f4cd391a5b74c7aa6cee9: Beginning new tablet copy session on tablet 98fcd7f5723d43448aa21d2095170c98 from peer 337d89be988d4793bc3f024a5e32096a at {username='slave'} at 127.0.0.1:53628: session id = 337d89be988d4793bc3f024a5e32096a-98fcd7f5723d43448aa21d2095170c98
I20251027 03:10:14.333405 1530 tablet_copy_source_session.cc:215] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:14.333690 1701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98fcd7f5723d43448aa21d2095170c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:14.334656 1701 tablet_copy_client.cc:806] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:14.334782 1701 tablet_copy_client.cc:670] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:14.335297 1701 tablet_copy_client.cc:538] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:14.335856 1701 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: Bootstrap starting.
I20251027 03:10:14.336858 1701 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:14.337018 1701 tablet_bootstrap.cc:492] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: Bootstrap complete.
I20251027 03:10:14.337095 1701 ts_tablet_manager.cc:1403] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:14.337214 1701 raft_consensus.cc:359] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.337275 1701 raft_consensus.cc:740] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 337d89be988d4793bc3f024a5e32096a, State: Initialized, Role: LEARNER
I20251027 03:10:14.337298 1701 consensus_queue.cc:260] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.337450 1701 ts_tablet_manager.cc:1434] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:14.337493 1683 heartbeater.cc:499] Master 127.27.151.126:36783 was elected leader, sending a full tablet report...
I20251027 03:10:14.337620 1530 tablet_copy_service.cc:342] P 0f0eeca3976f4cd391a5b74c7aa6cee9: Request end of tablet copy session 337d89be988d4793bc3f024a5e32096a-98fcd7f5723d43448aa21d2095170c98 received from {username='slave'} at 127.0.0.1:53628
I20251027 03:10:14.337680 1530 tablet_copy_service.cc:434] P 0f0eeca3976f4cd391a5b74c7aa6cee9: ending tablet copy session 337d89be988d4793bc3f024a5e32096a-98fcd7f5723d43448aa21d2095170c98 on tablet 98fcd7f5723d43448aa21d2095170c98 with peer 337d89be988d4793bc3f024a5e32096a
I20251027 03:10:14.484719 1701 ts_tablet_manager.cc:933] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: Initiating tablet copy from peer 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:14.484927 1701 tablet_copy_client.cc:323] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:41745
I20251027 03:10:14.485121 1530 tablet_copy_service.cc:140] P 0f0eeca3976f4cd391a5b74c7aa6cee9: Received BeginTabletCopySession request for tablet 85d903e62b144ce3aff41aed79cb38a1 from peer 337d89be988d4793bc3f024a5e32096a ({username='slave'} at 127.0.0.1:53628)
I20251027 03:10:14.485181 1530 tablet_copy_service.cc:161] P 0f0eeca3976f4cd391a5b74c7aa6cee9: Beginning new tablet copy session on tablet 85d903e62b144ce3aff41aed79cb38a1 from peer 337d89be988d4793bc3f024a5e32096a at {username='slave'} at 127.0.0.1:53628: session id = 337d89be988d4793bc3f024a5e32096a-85d903e62b144ce3aff41aed79cb38a1
I20251027 03:10:14.485579 1530 tablet_copy_source_session.cc:215] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:14.485857 1701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85d903e62b144ce3aff41aed79cb38a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:14.486763 1701 tablet_copy_client.cc:806] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:14.486869 1701 tablet_copy_client.cc:670] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:14.487375 1701 tablet_copy_client.cc:538] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:14.488132 1701 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: Bootstrap starting.
I20251027 03:10:14.489298 1701 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:14.489451 1701 tablet_bootstrap.cc:492] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: Bootstrap complete.
I20251027 03:10:14.489537 1701 ts_tablet_manager.cc:1403] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: Time spent bootstrapping tablet: real 0.001s user 0.002s sys 0.000s
I20251027 03:10:14.489665 1701 raft_consensus.cc:359] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.489727 1701 raft_consensus.cc:740] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 337d89be988d4793bc3f024a5e32096a, State: Initialized, Role: LEARNER
I20251027 03:10:14.489755 1701 consensus_queue.cc:260] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.489842 1701 ts_tablet_manager.cc:1434] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:14.490187 1530 tablet_copy_service.cc:342] P 0f0eeca3976f4cd391a5b74c7aa6cee9: Request end of tablet copy session 337d89be988d4793bc3f024a5e32096a-85d903e62b144ce3aff41aed79cb38a1 received from {username='slave'} at 127.0.0.1:53628
I20251027 03:10:14.490244 1530 tablet_copy_service.cc:434] P 0f0eeca3976f4cd391a5b74c7aa6cee9: ending tablet copy session 337d89be988d4793bc3f024a5e32096a-85d903e62b144ce3aff41aed79cb38a1 on tablet 85d903e62b144ce3aff41aed79cb38a1 with peer 337d89be988d4793bc3f024a5e32096a
I20251027 03:10:14.561719 1701 ts_tablet_manager.cc:933] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: Initiating tablet copy from peer 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:14.561916 1701 tablet_copy_client.cc:323] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:34825
I20251027 03:10:14.564205 1455 tablet_copy_service.cc:140] P 7771ba8de9194e7f9242c19e43d9064a: Received BeginTabletCopySession request for tablet 1e74f97f28b04ffd91c671ac78d0f66d from peer 337d89be988d4793bc3f024a5e32096a ({username='slave'} at 127.0.0.1:44166)
I20251027 03:10:14.564261 1455 tablet_copy_service.cc:161] P 7771ba8de9194e7f9242c19e43d9064a: Beginning new tablet copy session on tablet 1e74f97f28b04ffd91c671ac78d0f66d from peer 337d89be988d4793bc3f024a5e32096a at {username='slave'} at 127.0.0.1:44166: session id = 337d89be988d4793bc3f024a5e32096a-1e74f97f28b04ffd91c671ac78d0f66d
I20251027 03:10:14.564682 1455 tablet_copy_source_session.cc:215] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:14.564991 1701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e74f97f28b04ffd91c671ac78d0f66d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:14.565788 1701 tablet_copy_client.cc:806] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:14.565905 1701 tablet_copy_client.cc:670] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:14.566362 1701 tablet_copy_client.cc:538] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:14.566848 1701 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: Bootstrap starting.
I20251027 03:10:14.567720 1701 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:14.567847 1701 tablet_bootstrap.cc:492] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: Bootstrap complete.
I20251027 03:10:14.567940 1701 ts_tablet_manager.cc:1403] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.001s
I20251027 03:10:14.568037 1701 raft_consensus.cc:359] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.568089 1701 raft_consensus.cc:740] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 337d89be988d4793bc3f024a5e32096a, State: Initialized, Role: LEARNER
I20251027 03:10:14.568109 1701 consensus_queue.cc:260] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:14.568207 1701 ts_tablet_manager.cc:1434] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:14.568358 1455 tablet_copy_service.cc:342] P 7771ba8de9194e7f9242c19e43d9064a: Request end of tablet copy session 337d89be988d4793bc3f024a5e32096a-1e74f97f28b04ffd91c671ac78d0f66d received from {username='slave'} at 127.0.0.1:44166
I20251027 03:10:14.568398 1455 tablet_copy_service.cc:434] P 7771ba8de9194e7f9242c19e43d9064a: ending tablet copy session 337d89be988d4793bc3f024a5e32096a-1e74f97f28b04ffd91c671ac78d0f66d on tablet 1e74f97f28b04ffd91c671ac78d0f66d with peer 337d89be988d4793bc3f024a5e32096a
I20251027 03:10:14.761006 1697 raft_consensus.cc:493] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:14.761118 1697 raft_consensus.cc:515] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:14.761296 1697 leader_election.cc:290] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:14.761590 1520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91ce3a308ed84dac97f990e5b549788c" candidate_uuid: "7771ba8de9194e7f9242c19e43d9064a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" is_pre_election: true
I20251027 03:10:14.761704 1520 raft_consensus.cc:2468] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7771ba8de9194e7f9242c19e43d9064a in term 1.
I20251027 03:10:14.761852 1410 leader_election.cc:304] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, 7771ba8de9194e7f9242c19e43d9064a; no voters:
I20251027 03:10:14.761950 1697 raft_consensus.cc:2804] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251027 03:10:14.761952 1409 leader_election.cc:336] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
I20251027 03:10:14.762044 1697 raft_consensus.cc:493] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:14.762066 1697 raft_consensus.cc:3060] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:14.762665 1697 raft_consensus.cc:515] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:14.762768 1697 leader_election.cc:290] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 2 election: Requested vote from peers ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329), 0f0eeca3976f4cd391a5b74c7aa6cee9 (127.27.151.67:41745)
I20251027 03:10:14.762908 1520 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91ce3a308ed84dac97f990e5b549788c" candidate_uuid: "7771ba8de9194e7f9242c19e43d9064a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9"
I20251027 03:10:14.762975 1520 raft_consensus.cc:3060] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Advancing to term 2
W20251027 03:10:14.763331 1409 leader_election.cc:336] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
I20251027 03:10:14.763617 1520 raft_consensus.cc:2468] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7771ba8de9194e7f9242c19e43d9064a in term 2.
I20251027 03:10:14.763782 1410 leader_election.cc:304] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f0eeca3976f4cd391a5b74c7aa6cee9, 7771ba8de9194e7f9242c19e43d9064a; no voters: ba1f79e4d8aa4f7790466a6eafb6f065
I20251027 03:10:14.763885 1697 raft_consensus.cc:2804] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:10:14.763994 1697 raft_consensus.cc:697] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 LEADER]: Becoming Leader. State: Replica: 7771ba8de9194e7f9242c19e43d9064a, State: Running, Role: LEADER
I20251027 03:10:14.764086 1697 consensus_queue.cc:237] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } }
I20251027 03:10:14.764539 1264 catalog_manager.cc:5649] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a reported cstate change: term changed from 1 to 2, leader changed from ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) to 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66). New cstate: current_term: 2 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:14.951661 1657 raft_consensus.cc:1217] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251027 03:10:15.097568 1657 raft_consensus.cc:1217] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251027 03:10:15.177256 1657 raft_consensus.cc:1217] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251027 03:10:15.228530 1716 consensus_queue.cc:579] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.324s)
I20251027 03:10:15.276285 1520 raft_consensus.cc:1275] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251027 03:10:15.276531 1718 consensus_queue.cc:1048] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 03:10:15.278028 1409 consensus_peers.cc:597] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:15.284060 1716 consensus_queue.cc:579] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.336s)
W20251027 03:10:15.309387 1314 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver ba1f79e4d8aa4f7790466a6eafb6f065 not available for placement
I20251027 03:10:15.485980 1717 raft_consensus.cc:1064] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: attempting to promote NON_VOTER 337d89be988d4793bc3f024a5e32096a to VOTER
I20251027 03:10:15.486195 1717 consensus_queue.cc:237] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:15.486686 1657 raft_consensus.cc:1275] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:15.486686 1445 raft_consensus.cc:1275] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:15.486954 1723 consensus_queue.cc:1048] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:15.487052 1717 consensus_queue.cc:1048] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251027 03:10:15.487133 1484 consensus_peers.cc:597] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:15.487763 1717 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.487879 1445 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.488929 1657 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.488991 1264 catalog_manager.cc:5649] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:10:15.490072 1520 consensus_queue.cc:237] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:15.490398 1657 raft_consensus.cc:1275] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:15.490424 1445 raft_consensus.cc:1275] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:15.490602 1723 consensus_queue.cc:1048] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:15.490696 1693 consensus_queue.cc:1048] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:15.491168 1723 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.491266 1657 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.491914 1445 raft_consensus.cc:2955] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.492027 1263 catalog_manager.cc:5649] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a reported cstate change: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.492835 1251 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 98fcd7f5723d43448aa21d2095170c98 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20251027 03:10:15.498198 1250 catalog_manager.cc:4712] TS ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98fcd7f5723d43448aa21d2095170c98: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
I20251027 03:10:15.526392 1717 raft_consensus.cc:1064] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: attempting to promote NON_VOTER 337d89be988d4793bc3f024a5e32096a to VOTER
I20251027 03:10:15.526531 1717 consensus_queue.cc:237] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:15.526939 1445 raft_consensus.cc:1275] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:15.526939 1657 raft_consensus.cc:1275] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:15.527117 1716 consensus_queue.cc:1048] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:15.527202 1723 consensus_queue.cc:1048] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251027 03:10:15.527302 1484 consensus_peers.cc:597] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:15.527969 1717 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.528082 1657 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.528887 1445 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.528916 1265 catalog_manager.cc:5649] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:10:15.529630 1520 consensus_queue.cc:237] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:15.529953 1445 raft_consensus.cc:1275] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:15.529961 1657 raft_consensus.cc:1275] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Refusing update from remote peer 0f0eeca3976f4cd391a5b74c7aa6cee9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:15.530112 1693 consensus_queue.cc:1048] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:15.530179 1723 consensus_queue.cc:1048] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:15.530658 1716 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.530782 1657 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.530767 1445 raft_consensus.cc:2955] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.531528 1251 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 85d903e62b144ce3aff41aed79cb38a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:15.531502 1264 catalog_manager.cc:5649] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New cstate: current_term: 1 leader_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:10:15.648872 1718 consensus_queue.cc:579] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.527s)
I20251027 03:10:15.735383 1718 raft_consensus.cc:1064] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: attempting to promote NON_VOTER 337d89be988d4793bc3f024a5e32096a to VOTER
I20251027 03:10:15.735560 1718 consensus_queue.cc:237] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:15.736069 1657 raft_consensus.cc:1275] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 LEARNER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:15.736066 1520 raft_consensus.cc:1275] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:15.736320 1735 consensus_queue.cc:1048] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:15.736461 1711 consensus_queue.cc:1048] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251027 03:10:15.736567 1409 consensus_peers.cc:597] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:15.738611 1711 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.738781 1657 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.739616 1520 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.739735 1265 catalog_manager.cc:5649] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a reported cstate change: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.740698 1445 consensus_queue.cc:237] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:15.741075 1657 raft_consensus.cc:1275] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:15.741096 1520 raft_consensus.cc:1275] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:15.741298 1718 consensus_queue.cc:1048] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:15.741503 1735 consensus_queue.cc:1048] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:15.742026 1718 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.742173 1657 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.742323 1520 raft_consensus.cc:2955] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:15.743273 1250 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 1e74f97f28b04ffd91c671ac78d0f66d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:15.743474 1264 catalog_manager.cc:5649] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 reported cstate change: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New cstate: current_term: 1 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:16.206593 1735 consensus_queue.cc:579] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Leader has been unable to successfully communicate with peer ba1f79e4d8aa4f7790466a6eafb6f065 for more than 1 seconds (1.442s)
I20251027 03:10:16.207437 1445 consensus_queue.cc:237] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:16.207979 1520 raft_consensus.cc:1275] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251027 03:10:16.208094 1410 consensus_peers.cc:597] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a -> Peer 337d89be988d4793bc3f024a5e32096a (127.27.151.68:44013): Couldn't send request to peer 337d89be988d4793bc3f024a5e32096a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91ce3a308ed84dac97f990e5b549788c. This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:16.208109 1409 consensus_peers.cc:597] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:16.208214 1735 consensus_queue.cc:1048] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:10:16.208817 1711 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:16.209757 1250 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 91ce3a308ed84dac97f990e5b549788c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251027 03:10:16.209781 1520 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } } }
I20251027 03:10:16.209865 1265 catalog_manager.cc:5649] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a reported cstate change: config changed from index -1 to 2, NON_VOTER 337d89be988d4793bc3f024a5e32096a (127.27.151.68) added. New cstate: current_term: 2 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251027 03:10:16.309597 1314 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver ba1f79e4d8aa4f7790466a6eafb6f065 not available for placement
I20251027 03:10:16.585305 1738 ts_tablet_manager.cc:933] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: Initiating tablet copy from peer 7771ba8de9194e7f9242c19e43d9064a (127.27.151.66:34825)
I20251027 03:10:16.585480 1738 tablet_copy_client.cc:323] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:34825
I20251027 03:10:16.585700 1455 tablet_copy_service.cc:140] P 7771ba8de9194e7f9242c19e43d9064a: Received BeginTabletCopySession request for tablet 91ce3a308ed84dac97f990e5b549788c from peer 337d89be988d4793bc3f024a5e32096a ({username='slave'} at 127.0.0.1:44166)
I20251027 03:10:16.585748 1455 tablet_copy_service.cc:161] P 7771ba8de9194e7f9242c19e43d9064a: Beginning new tablet copy session on tablet 91ce3a308ed84dac97f990e5b549788c from peer 337d89be988d4793bc3f024a5e32096a at {username='slave'} at 127.0.0.1:44166: session id = 337d89be988d4793bc3f024a5e32096a-91ce3a308ed84dac97f990e5b549788c
I20251027 03:10:16.586194 1455 tablet_copy_source_session.cc:215] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:16.586501 1738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91ce3a308ed84dac97f990e5b549788c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:16.587388 1738 tablet_copy_client.cc:806] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:16.587492 1738 tablet_copy_client.cc:670] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:16.588043 1738 tablet_copy_client.cc:538] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:16.588627 1738 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: Bootstrap starting.
I20251027 03:10:16.589619 1738 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:16.589823 1738 tablet_bootstrap.cc:492] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: Bootstrap complete.
I20251027 03:10:16.589905 1738 ts_tablet_manager.cc:1403] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:16.590027 1738 raft_consensus.cc:359] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:16.590085 1738 raft_consensus.cc:740] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 337d89be988d4793bc3f024a5e32096a, State: Initialized, Role: LEARNER
I20251027 03:10:16.590112 1738 consensus_queue.cc:260] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: true } }
I20251027 03:10:16.590246 1738 ts_tablet_manager.cc:1434] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:16.590430 1455 tablet_copy_service.cc:342] P 7771ba8de9194e7f9242c19e43d9064a: Request end of tablet copy session 337d89be988d4793bc3f024a5e32096a-91ce3a308ed84dac97f990e5b549788c received from {username='slave'} at 127.0.0.1:44166
I20251027 03:10:16.590471 1455 tablet_copy_service.cc:434] P 7771ba8de9194e7f9242c19e43d9064a: ending tablet copy session 337d89be988d4793bc3f024a5e32096a-91ce3a308ed84dac97f990e5b549788c on tablet 91ce3a308ed84dac97f990e5b549788c with peer 337d89be988d4793bc3f024a5e32096a
W20251027 03:10:16.683749 1250 catalog_manager.cc:4712] TS ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98fcd7f5723d43448aa21d2095170c98: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
I20251027 03:10:16.966450 1657 raft_consensus.cc:1217] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20251027 03:10:17.309827 1314 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver ba1f79e4d8aa4f7790466a6eafb6f065 not available for placement
I20251027 03:10:17.382419 1735 raft_consensus.cc:1064] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: attempting to promote NON_VOTER 337d89be988d4793bc3f024a5e32096a to VOTER
I20251027 03:10:17.382658 1735 consensus_queue.cc:237] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:17.383185 1520 raft_consensus.cc:1275] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:10:17.383262 1657 raft_consensus.cc:1275] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 LEARNER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:10:17.383481 1746 consensus_queue.cc:1048] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:17.383661 1747 consensus_queue.cc:1048] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251027 03:10:17.383807 1409 consensus_peers.cc:597] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a -> Peer ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): Couldn't send request to peer ba1f79e4d8aa4f7790466a6eafb6f065. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:17.384402 1747 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.384536 1657 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.385433 1520 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.385603 1263 catalog_manager.cc:5649] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a reported cstate change: config changed from index 2 to 3, 337d89be988d4793bc3f024a5e32096a (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ba1f79e4d8aa4f7790466a6eafb6f065" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 45329 } } peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.387153 1445 consensus_queue.cc:237] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } }
I20251027 03:10:17.387481 1520 raft_consensus.cc:1275] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251027 03:10:17.387573 1657 raft_consensus.cc:1275] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 FOLLOWER]: Refusing update from remote peer 7771ba8de9194e7f9242c19e43d9064a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251027 03:10:17.387641 1746 consensus_queue.cc:1048] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:17.387727 1747 consensus_queue.cc:1048] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [LEADER]: Connected to new peer: Peer: permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:17.388231 1746 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.388355 1657 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.388381 1520 raft_consensus.cc:2955] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } } }
I20251027 03:10:17.389235 1250 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 91ce3a308ed84dac97f990e5b549788c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:17.389214 1265 catalog_manager.cc:5649] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a reported cstate change: config changed from index 3 to 4, VOTER ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65) evicted. New cstate: current_term: 2 leader_uuid: "7771ba8de9194e7f9242c19e43d9064a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f0eeca3976f4cd391a5b74c7aa6cee9" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7771ba8de9194e7f9242c19e43d9064a" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337d89be988d4793bc3f024a5e32096a" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 44013 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251027 03:10:17.701391 1250 catalog_manager.cc:4712] TS ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 91ce3a308ed84dac97f990e5b549788c: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
W20251027 03:10:17.958429 1250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20251027 03:10:19.561280 1250 catalog_manager.cc:4712] TS ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 91ce3a308ed84dac97f990e5b549788c: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
I20251027 03:10:20.908967 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:10:20.911638 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:20.911756 28253 tablet_replica.cc:333] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a: stopping tablet replica
I20251027 03:10:20.911808 28253 raft_consensus.cc:2243] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.911866 28253 raft_consensus.cc:2272] T 98fcd7f5723d43448aa21d2095170c98 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.912029 28253 tablet_replica.cc:333] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a: stopping tablet replica
I20251027 03:10:20.912065 28253 raft_consensus.cc:2243] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:20.912117 28253 raft_consensus.cc:2272] T 1e74f97f28b04ffd91c671ac78d0f66d P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.912254 28253 tablet_replica.cc:333] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a: stopping tablet replica
I20251027 03:10:20.912290 28253 raft_consensus.cc:2243] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.912341 28253 raft_consensus.cc:2272] T 85d903e62b144ce3aff41aed79cb38a1 P 7771ba8de9194e7f9242c19e43d9064a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.912451 28253 tablet_replica.cc:333] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a: stopping tablet replica
I20251027 03:10:20.912482 28253 raft_consensus.cc:2243] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:10:20.912531 28253 raft_consensus.cc:2272] T 91ce3a308ed84dac97f990e5b549788c P 7771ba8de9194e7f9242c19e43d9064a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.924064 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:10:20.925092 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:10:20.927771 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:20.927874 28253 tablet_replica.cc:333] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9: stopping tablet replica
I20251027 03:10:20.927932 28253 raft_consensus.cc:2243] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:20.927992 28253 raft_consensus.cc:2272] T 98fcd7f5723d43448aa21d2095170c98 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.928128 28253 tablet_replica.cc:333] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9: stopping tablet replica
I20251027 03:10:20.928189 28253 raft_consensus.cc:2243] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.928252 28253 raft_consensus.cc:2272] T 91ce3a308ed84dac97f990e5b549788c P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.928360 28253 tablet_replica.cc:333] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9: stopping tablet replica
I20251027 03:10:20.928391 28253 raft_consensus.cc:2243] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:20.928442 28253 raft_consensus.cc:2272] T 85d903e62b144ce3aff41aed79cb38a1 P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.928545 28253 tablet_replica.cc:333] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9: stopping tablet replica
I20251027 03:10:20.928578 28253 raft_consensus.cc:2243] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.928625 28253 raft_consensus.cc:2272] T 1e74f97f28b04ffd91c671ac78d0f66d P 0f0eeca3976f4cd391a5b74c7aa6cee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.940065 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:10:20.941112 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:10:20.943181 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:20.943265 28253 tablet_replica.cc:333] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a: stopping tablet replica
I20251027 03:10:20.943317 28253 raft_consensus.cc:2243] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.943359 28253 raft_consensus.cc:2272] T 91ce3a308ed84dac97f990e5b549788c P 337d89be988d4793bc3f024a5e32096a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.943475 28253 tablet_replica.cc:333] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a: stopping tablet replica
I20251027 03:10:20.943518 28253 raft_consensus.cc:2243] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.943547 28253 raft_consensus.cc:2272] T 98fcd7f5723d43448aa21d2095170c98 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.943661 28253 tablet_replica.cc:333] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a: stopping tablet replica
I20251027 03:10:20.943706 28253 raft_consensus.cc:2243] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.943729 28253 raft_consensus.cc:2272] T 1e74f97f28b04ffd91c671ac78d0f66d P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.943801 28253 tablet_replica.cc:333] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a: stopping tablet replica
I20251027 03:10:20.943823 28253 raft_consensus.cc:2243] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:20.943851 28253 raft_consensus.cc:2272] T 85d903e62b144ce3aff41aed79cb38a1 P 337d89be988d4793bc3f024a5e32096a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:20.955044 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:10:20.955884 28253 master.cc:561] Master@127.27.151.126:36783 shutting down...
W20251027 03:10:21.650143 1250 catalog_manager.cc:4712] TS ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 91ce3a308ed84dac97f990e5b549788c: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
W20251027 03:10:23.978839 1250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20251027 03:10:23.979329 1250 catalog_manager.cc:4712] TS ba1f79e4d8aa4f7790466a6eafb6f065 (127.27.151.65:45329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 98fcd7f5723d43448aa21d2095170c98: Network error: Client connection negotiation failed: client connection to 127.27.151.65:45329: connect: Connection refused (error 111)
I20251027 03:10:24.548583 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:24.548714 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:24.548739 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2b7d56cf207e438f8b7fc7ccbfa7f157: stopping tablet replica
I20251027 03:10:24.560392 28253 master.cc:583] Master@127.27.151.126:36783 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13275 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20251027 03:10:24.566505 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:36947
I20251027 03:10:24.566658 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:24.568006 1763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:24.568094 1762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.568202 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:24.568208 1765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.568456 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:24.568493 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:24.568512 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534624568512 us; error 0 us; skew 500 ppm
I20251027 03:10:24.569038 28253 webserver.cc:492] Webserver started at http://127.27.151.126:32977/ using document root <none> and password file <none>
I20251027 03:10:24.569115 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:24.569159 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:24.569211 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:24.569450 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "8f5561234ce244118f40ffc2d62d87b7"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.570241 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:24.570634 1770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.570767 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.570811 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "8f5561234ce244118f40ffc2d62d87b7"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.570845 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:24.578989 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:24.579118 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:24.581369 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:36947
I20251027 03:10:24.584471 1822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:36947 every 8 connection(s)
I20251027 03:10:24.584599 1823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:24.585491 1823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7: Bootstrap starting.
I20251027 03:10:24.585731 1823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:24.586125 1823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7: No bootstrap required, opened a new log
I20251027 03:10:24.586223 1823 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5561234ce244118f40ffc2d62d87b7" member_type: VOTER }
I20251027 03:10:24.586262 1823 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:24.586275 1823 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5561234ce244118f40ffc2d62d87b7, State: Initialized, Role: FOLLOWER
I20251027 03:10:24.586308 1823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5561234ce244118f40ffc2d62d87b7" member_type: VOTER }
I20251027 03:10:24.586351 1823 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:10:24.586372 1823 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:10:24.586386 1823 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:24.586828 1823 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5561234ce244118f40ffc2d62d87b7" member_type: VOTER }
I20251027 03:10:24.586876 1823 leader_election.cc:304] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5561234ce244118f40ffc2d62d87b7; no voters:
I20251027 03:10:24.586961 1823 leader_election.cc:290] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:10:24.587014 1826 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:24.587076 1823 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:10:24.587138 1826 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 1 LEADER]: Becoming Leader. State: Replica: 8f5561234ce244118f40ffc2d62d87b7, State: Running, Role: LEADER
I20251027 03:10:24.587188 1826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5561234ce244118f40ffc2d62d87b7" member_type: VOTER }
I20251027 03:10:24.587347 1826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5561234ce244118f40ffc2d62d87b7. Latest consensus state: current_term: 1 leader_uuid: "8f5561234ce244118f40ffc2d62d87b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5561234ce244118f40ffc2d62d87b7" member_type: VOTER } }
I20251027 03:10:24.587366 1827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5561234ce244118f40ffc2d62d87b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5561234ce244118f40ffc2d62d87b7" member_type: VOTER } }
I20251027 03:10:24.587427 1826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [sys.catalog]: This master's current role is: LEADER
I20251027 03:10:24.587494 1827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [sys.catalog]: This master's current role is: LEADER
I20251027 03:10:24.587711 1833 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:10:24.587831 1833 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:10:24.588330 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:10:24.588356 1833 catalog_manager.cc:1357] Generated new cluster ID: 6a4a1af423734307bf4520fed7987c37
I20251027 03:10:24.588390 1833 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:10:24.599922 1833 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:10:24.600328 1833 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:10:24.603825 1833 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7: Generated new TSK 0
I20251027 03:10:24.603886 1833 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:10:24.620076 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:24.621251 1845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:24.621232 1844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.621289 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:24.621291 1847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.621541 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:24.621575 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:24.621588 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534624621589 us; error 0 us; skew 500 ppm
I20251027 03:10:24.622112 28253 webserver.cc:492] Webserver started at http://127.27.151.65:40825/ using document root <none> and password file <none>
I20251027 03:10:24.622179 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:24.622228 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:24.622277 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:24.622500 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "598c4e01f9244c27bb5c87ced4782621"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.623270 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:24.623670 1852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.623807 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.623842 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "598c4e01f9244c27bb5c87ced4782621"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.623878 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:24.637737 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:24.637887 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:24.638017 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:24.638185 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:24.638211 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.638239 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:24.638266 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.640942 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:35921
I20251027 03:10:24.640982 1915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:35921 every 8 connection(s)
I20251027 03:10:24.641270 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:24.642234 1921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:24.642249 1920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:24.642423 1923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.642568 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:24.642674 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:24.642704 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:24.642719 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534624642720 us; error 0 us; skew 500 ppm
I20251027 03:10:24.643229 28253 webserver.cc:492] Webserver started at http://127.27.151.66:38869/ using document root <none> and password file <none>
I20251027 03:10:24.643297 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:24.643340 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:24.643385 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:24.643596 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "d15b3c083d1441b79a054ba072903211"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.644609 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:24.644793 1916 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:24.644856 1916 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:24.644966 1916 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:24.645087 1928 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.645172 1787 ts_manager.cc:194] Registered new tserver with Master: 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921)
I20251027 03:10:24.645180 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.645234 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "d15b3c083d1441b79a054ba072903211"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.645273 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:24.645958 1787 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45500
I20251027 03:10:24.668994 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:24.669144 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:24.669242 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:24.669379 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:24.669401 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.669421 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:24.669438 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.672017 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:40215
I20251027 03:10:24.672055 1991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:40215 every 8 connection(s)
I20251027 03:10:24.672356 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:24.673327 1995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:24.673374 1996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.673839 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:24.673400 1998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:24.674019 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:24.674062 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:24.674074 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534624674075 us; error 0 us; skew 500 ppm
I20251027 03:10:24.674594 28253 webserver.cc:492] Webserver started at http://127.27.151.67:39523/ using document root <none> and password file <none>
I20251027 03:10:24.674670 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:24.674719 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:24.674768 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:24.675061 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "39262c5b16884c76b3674e96334e5d72"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.675169 1992 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:24.675226 1992 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:24.675321 1992 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:24.675499 1787 ts_manager.cc:194] Registered new tserver with Master: d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215)
I20251027 03:10:24.675797 1787 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45516
I20251027 03:10:24.676074 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:24.676504 2003 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.676615 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.676654 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "39262c5b16884c76b3674e96334e5d72"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:24.676698 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:24.683696 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:24.683854 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:24.683979 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:24.684131 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:24.684183 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.684219 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:24.684247 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:24.687137 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:41635
I20251027 03:10:24.687196 2066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:41635 every 8 connection(s)
I20251027 03:10:24.689694 2067 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:24.689746 2067 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:24.689831 2067 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:24.690042 1787 ts_manager.cc:194] Registered new tserver with Master: 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:24.690341 1787 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45518
I20251027 03:10:24.690578 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003265348s
I20251027 03:10:25.646567 1916 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
I20251027 03:10:25.676371 1992 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
I20251027 03:10:25.690894 2067 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
I20251027 03:10:25.715358 28253 test_util.cc:276] Using random seed: 1033901209
I20251027 03:10:25.724292 1787 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45522:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:10:25.724424 1787 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:10:25.729059 1880 tablet_service.cc:1505] Processing CreateTablet for tablet a8553ab4d14b48d3a8da81572d5e2882 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:25.729105 1877 tablet_service.cc:1505] Processing CreateTablet for tablet 0dc9f313b28b48d2ae0711f6ede99030 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:25.729197 1880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8553ab4d14b48d3a8da81572d5e2882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.729076 1878 tablet_service.cc:1505] Processing CreateTablet for tablet 606568a1268245b3839b89f59239ef13 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:25.729379 1878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606568a1268245b3839b89f59239ef13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.729835 2031 tablet_service.cc:1505] Processing CreateTablet for tablet a8553ab4d14b48d3a8da81572d5e2882 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:25.729928 2031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8553ab4d14b48d3a8da81572d5e2882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.730367 1956 tablet_service.cc:1505] Processing CreateTablet for tablet a8553ab4d14b48d3a8da81572d5e2882 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:25.730459 1956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8553ab4d14b48d3a8da81572d5e2882. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.729069 1879 tablet_service.cc:1505] Processing CreateTablet for tablet 836f565450b04d548ab63bc15a22a254 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:25.730748 2030 tablet_service.cc:1505] Processing CreateTablet for tablet 836f565450b04d548ab63bc15a22a254 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:25.730777 1879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 836f565450b04d548ab63bc15a22a254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.730832 2030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 836f565450b04d548ab63bc15a22a254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.731784 1955 tablet_service.cc:1505] Processing CreateTablet for tablet 836f565450b04d548ab63bc15a22a254 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:25.731786 2089 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:25.731876 1955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 836f565450b04d548ab63bc15a22a254. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.731998 2029 tablet_service.cc:1505] Processing CreateTablet for tablet 606568a1268245b3839b89f59239ef13 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:25.732086 2029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606568a1268245b3839b89f59239ef13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.732303 2089 tablet_bootstrap.cc:654] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.732323 2028 tablet_service.cc:1505] Processing CreateTablet for tablet 0dc9f313b28b48d2ae0711f6ede99030 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:25.732407 2028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc9f313b28b48d2ae0711f6ede99030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.732990 1954 tablet_service.cc:1505] Processing CreateTablet for tablet 606568a1268245b3839b89f59239ef13 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:25.733065 1954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 606568a1268245b3839b89f59239ef13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.733338 2089 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: No bootstrap required, opened a new log
I20251027 03:10:25.733369 2089 ts_tablet_manager.cc:1403] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:10:25.733484 2089 raft_consensus.cc:359] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.733542 2089 raft_consensus.cc:385] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.733561 2089 raft_consensus.cc:740] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.733603 2089 consensus_queue.cc:260] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.733711 1877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc9f313b28b48d2ae0711f6ede99030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.733769 2089 ts_tablet_manager.cc:1434] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:25.733860 2089 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:25.734187 1953 tablet_service.cc:1505] Processing CreateTablet for tablet 0dc9f313b28b48d2ae0711f6ede99030 (DEFAULT_TABLE table=test-workload [id=b38cbcc18f604c9aa65643c564b769eb]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:25.734277 1953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc9f313b28b48d2ae0711f6ede99030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:25.734855 2087 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:25.735234 2087 tablet_bootstrap.cc:654] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.735543 2088 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:25.735716 2087 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: No bootstrap required, opened a new log
I20251027 03:10:25.735745 2087 ts_tablet_manager.cc:1403] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:25.735848 2087 raft_consensus.cc:359] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.734213 2089 tablet_bootstrap.cc:654] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.735946 2088 tablet_bootstrap.cc:654] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.735937 2087 raft_consensus.cc:385] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.736037 2087 raft_consensus.cc:740] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.736187 2087 consensus_queue.cc:260] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.736358 2087 ts_tablet_manager.cc:1434] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:25.736413 2087 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:25.736667 2088 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: No bootstrap required, opened a new log
I20251027 03:10:25.736711 2088 ts_tablet_manager.cc:1403] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:25.736764 2087 tablet_bootstrap.cc:654] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.736799 2089 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: No bootstrap required, opened a new log
I20251027 03:10:25.736828 2089 ts_tablet_manager.cc:1403] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:10:25.736846 2088 raft_consensus.cc:359] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.736912 2088 raft_consensus.cc:385] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.736929 2088 raft_consensus.cc:740] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.736938 2089 raft_consensus.cc:359] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.736970 2088 consensus_queue.cc:260] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.736979 2089 raft_consensus.cc:385] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.737036 2089 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.737087 2089 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.737114 2088 ts_tablet_manager.cc:1434] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.737164 2088 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:25.737169 2089 ts_tablet_manager.cc:1434] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.737244 2089 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:25.737368 2087 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: No bootstrap required, opened a new log
I20251027 03:10:25.737398 2087 ts_tablet_manager.cc:1403] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:25.737540 2088 tablet_bootstrap.cc:654] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.737591 2089 tablet_bootstrap.cc:654] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.737531 2087 raft_consensus.cc:359] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.737928 2087 raft_consensus.cc:385] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.737985 2087 raft_consensus.cc:740] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.738052 2087 consensus_queue.cc:260] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.738114 2088 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: No bootstrap required, opened a new log
I20251027 03:10:25.738134 2087 ts_tablet_manager.cc:1434] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:25.738165 2088 ts_tablet_manager.cc:1403] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:25.738183 2087 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:25.738330 2088 raft_consensus.cc:359] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.738391 2088 raft_consensus.cc:385] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.738412 2088 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.738456 2088 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.738528 2088 ts_tablet_manager.cc:1434] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.738577 2088 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:25.738579 2089 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: No bootstrap required, opened a new log
I20251027 03:10:25.738677 2089 ts_tablet_manager.cc:1403] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:25.738837 2087 tablet_bootstrap.cc:654] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.738806 2089 raft_consensus.cc:359] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.738873 2088 tablet_bootstrap.cc:654] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.738880 2089 raft_consensus.cc:385] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.738933 2089 raft_consensus.cc:740] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.738981 2089 consensus_queue.cc:260] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.739069 2089 ts_tablet_manager.cc:1434] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.739111 2089 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:25.739507 2089 tablet_bootstrap.cc:654] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.739568 2088 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: No bootstrap required, opened a new log
I20251027 03:10:25.739614 2088 ts_tablet_manager.cc:1403] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:25.739784 2088 raft_consensus.cc:359] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.739840 2088 raft_consensus.cc:385] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.739863 2088 raft_consensus.cc:740] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.739907 2088 consensus_queue.cc:260] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.739985 2088 ts_tablet_manager.cc:1434] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.740034 2088 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:25.740466 2088 tablet_bootstrap.cc:654] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.740921 2087 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: No bootstrap required, opened a new log
I20251027 03:10:25.740942 2089 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: No bootstrap required, opened a new log
I20251027 03:10:25.740964 2087 ts_tablet_manager.cc:1403] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:10:25.740972 2089 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20251027 03:10:25.741108 2087 raft_consensus.cc:359] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.741159 2087 raft_consensus.cc:385] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.741181 2087 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.741235 2087 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.741183 2089 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.741283 2089 raft_consensus.cc:385] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.741307 2089 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.741319 2087 ts_tablet_manager.cc:1434] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.741369 2087 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:25.741452 2089 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.741576 2089 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:25.741782 2087 tablet_bootstrap.cc:654] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:25.742254 2088 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: No bootstrap required, opened a new log
I20251027 03:10:25.742303 2088 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:10:25.742363 2087 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: No bootstrap required, opened a new log
I20251027 03:10:25.742409 2087 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:25.742491 2088 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.742555 2088 raft_consensus.cc:385] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.742573 2088 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.742611 2087 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.742660 2087 raft_consensus.cc:385] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:25.742710 2088 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.742863 2088 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:25.742710 2087 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:25.742976 2087 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.743044 2087 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:25.836057 2091 raft_consensus.cc:493] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:25.836177 2091 raft_consensus.cc:515] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.836365 2091 leader_election.cc:290] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:25.839666 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:25.839670 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:25.839788 1890 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 0.
I20251027 03:10:25.839794 2041 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 0.
I20251027 03:10:25.839952 1929 leader_election.cc:304] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:25.840045 2091 raft_consensus.cc:2804] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:25.840102 2091 raft_consensus.cc:493] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:25.840128 2091 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:25.842622 2091 raft_consensus.cc:515] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.842743 2091 leader_election.cc:290] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:25.842892 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:25.842939 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:25.842947 1890 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:25.843029 2041 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:25.843557 1890 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:25.843660 2041 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:25.843716 1929 leader_election.cc:304] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:25.843787 2091 raft_consensus.cc:2804] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:25.843901 2091 raft_consensus.cc:697] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: Becoming Leader. State: Replica: d15b3c083d1441b79a054ba072903211, State: Running, Role: LEADER
I20251027 03:10:25.843948 2091 consensus_queue.cc:237] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.844208 2095 raft_consensus.cc:493] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:25.844262 2095 raft_consensus.cc:515] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.844400 2095 leader_election.cc:290] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:25.844442 1785 catalog_manager.cc:5649] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 reported cstate change: term changed from 0 to 1, leader changed from <none> to d15b3c083d1441b79a054ba072903211 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "d15b3c083d1441b79a054ba072903211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:25.846678 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211" is_pre_election: true
I20251027 03:10:25.846719 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:25.846732 1966 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 0.
I20251027 03:10:25.846796 2041 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 0.
I20251027 03:10:25.846884 1854 leader_election.cc:304] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:25.846968 2095 raft_consensus.cc:2804] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:25.846997 2095 raft_consensus.cc:493] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:25.847018 2095 raft_consensus.cc:3060] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:25.847445 2095 raft_consensus.cc:515] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.847554 2095 leader_election.cc:290] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 1 election: Requested vote from peers d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:25.847661 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211"
I20251027 03:10:25.847707 1966 raft_consensus.cc:3060] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:25.847666 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:25.847743 2041 raft_consensus.cc:3060] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:25.848170 1966 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 1.
I20251027 03:10:25.848230 2041 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 1.
I20251027 03:10:25.848315 1854 leader_election.cc:304] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:25.848384 2095 raft_consensus.cc:2804] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:25.848500 2095 raft_consensus.cc:697] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 LEADER]: Becoming Leader. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Running, Role: LEADER
I20251027 03:10:25.848536 2095 consensus_queue.cc:237] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:25.848918 1785 catalog_manager.cc:5649] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 reported cstate change: term changed from 0 to 1, leader changed from <none> to 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65). New cstate: current_term: 1 leader_uuid: "598c4e01f9244c27bb5c87ced4782621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
W20251027 03:10:25.891974 1917 tablet.cc:2378] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 03:10:26.004699 2091 raft_consensus.cc:493] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:26.004802 2091 raft_consensus.cc:515] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.004997 2091 leader_election.cc:290] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:26.005194 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:26.005237 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:26.005268 1890 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 0.
I20251027 03:10:26.005362 2041 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 0.
I20251027 03:10:26.005470 1929 leader_election.cc:304] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:26.005559 2091 raft_consensus.cc:2804] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:26.005599 2091 raft_consensus.cc:493] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:26.005618 2091 raft_consensus.cc:3060] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:26.006220 2091 raft_consensus.cc:515] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.006307 2091 leader_election.cc:290] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:26.006495 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:26.006505 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:26.006569 1890 raft_consensus.cc:3060] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:26.006570 2041 raft_consensus.cc:3060] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:26.007182 1890 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:26.007184 2041 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:26.007360 1929 leader_election.cc:304] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:26.007441 2091 raft_consensus.cc:2804] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:26.007491 2091 raft_consensus.cc:697] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: Becoming Leader. State: Replica: d15b3c083d1441b79a054ba072903211, State: Running, Role: LEADER
I20251027 03:10:26.007524 2091 consensus_queue.cc:237] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.008083 1785 catalog_manager.cc:5649] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 reported cstate change: term changed from 0 to 1, leader changed from <none> to d15b3c083d1441b79a054ba072903211 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "d15b3c083d1441b79a054ba072903211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:26.088861 2097 raft_consensus.cc:493] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:26.088930 2091 raft_consensus.cc:493] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:26.088991 2091 raft_consensus.cc:515] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.088981 2097 raft_consensus.cc:515] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.089156 2091 leader_election.cc:290] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:26.089215 2097 leader_election.cc:290] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215)
I20251027 03:10:26.089354 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:26.089426 1890 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 0.
I20251027 03:10:26.089574 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:26.089641 2041 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 0.
I20251027 03:10:26.089800 1929 leader_election.cc:304] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:26.089908 2091 raft_consensus.cc:2804] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:26.089944 2091 raft_consensus.cc:493] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:26.089967 2091 raft_consensus.cc:3060] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:26.090759 2091 raft_consensus.cc:515] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.090862 2091 leader_election.cc:290] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:26.091061 2041 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:26.091061 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:26.091140 2041 raft_consensus.cc:3060] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:26.091145 1890 raft_consensus.cc:3060] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:26.091747 2041 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:26.091747 1890 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:26.091945 1929 leader_election.cc:304] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:26.092325 1890 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "39262c5b16884c76b3674e96334e5d72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:26.092386 1890 raft_consensus.cc:2393] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39262c5b16884c76b3674e96334e5d72 in current term 1: Already voted for candidate d15b3c083d1441b79a054ba072903211 in this term.
I20251027 03:10:26.092505 2091 raft_consensus.cc:2804] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:26.092577 2091 raft_consensus.cc:697] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: Becoming Leader. State: Replica: d15b3c083d1441b79a054ba072903211, State: Running, Role: LEADER
I20251027 03:10:26.092618 1966 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "39262c5b16884c76b3674e96334e5d72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211" is_pre_election: true
I20251027 03:10:26.092633 2091 consensus_queue.cc:237] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:26.092814 2005 leader_election.cc:304] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72; no voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211
I20251027 03:10:26.092900 2097 raft_consensus.cc:2749] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 03:10:26.093225 1785 catalog_manager.cc:5649] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 reported cstate change: term changed from 0 to 1, leader changed from <none> to d15b3c083d1441b79a054ba072903211 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "d15b3c083d1441b79a054ba072903211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:26.177438 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:10:26.180034 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:26.180240 28253 tablet_replica.cc:333] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: stopping tablet replica
I20251027 03:10:26.180306 28253 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:26.180363 28253 pending_rounds.cc:70] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Trying to abort 1 pending ops.
I20251027 03:10:26.180389 28253 pending_rounds.cc:77] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245827475890176 op_type: NO_OP noop_request { }
I20251027 03:10:26.180442 28253 raft_consensus.cc:2889] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:26.180470 28253 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.180742 28253 tablet_replica.cc:333] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: stopping tablet replica
I20251027 03:10:26.180792 28253 raft_consensus.cc:2243] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.180816 28253 raft_consensus.cc:2272] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.180975 28253 tablet_replica.cc:333] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: stopping tablet replica
I20251027 03:10:26.181007 28253 raft_consensus.cc:2243] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.181039 28253 raft_consensus.cc:2272] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.181294 28253 tablet_replica.cc:333] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: stopping tablet replica
I20251027 03:10:26.181342 28253 raft_consensus.cc:2243] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.181368 28253 raft_consensus.cc:2272] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.193109 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:10:26.193917 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:10:26.196552 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:26.196678 28253 tablet_replica.cc:333] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: stopping tablet replica
I20251027 03:10:26.196748 28253 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.196779 28253 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.196936 28253 tablet_replica.cc:333] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: stopping tablet replica
I20251027 03:10:26.196969 28253 raft_consensus.cc:2243] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:26.197000 28253 pending_rounds.cc:70] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Trying to abort 1 pending ops.
I20251027 03:10:26.197014 28253 pending_rounds.cc:77] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245828476039168 op_type: NO_OP noop_request { }
I20251027 03:10:26.197048 28253 raft_consensus.cc:2889] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:26.197070 28253 raft_consensus.cc:2272] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.197265 28253 tablet_replica.cc:333] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: stopping tablet replica
I20251027 03:10:26.197316 28253 raft_consensus.cc:2243] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:26.197347 28253 pending_rounds.cc:70] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Trying to abort 1 pending ops.
I20251027 03:10:26.197363 28253 pending_rounds.cc:77] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245828127404032 op_type: NO_OP noop_request { }
I20251027 03:10:26.197391 28253 raft_consensus.cc:2889] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:26.197405 28253 raft_consensus.cc:2272] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.197600 28253 tablet_replica.cc:333] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: stopping tablet replica
I20251027 03:10:26.197634 28253 raft_consensus.cc:2243] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:26.197669 28253 pending_rounds.cc:70] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Trying to abort 1 pending ops.
I20251027 03:10:26.197686 28253 pending_rounds.cc:77] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245827457282048 op_type: NO_OP noop_request { }
I20251027 03:10:26.197716 28253 raft_consensus.cc:2889] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:26.197731 28253 raft_consensus.cc:2272] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.209355 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:10:26.210219 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:10:26.212599 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:26.212777 28253 tablet_replica.cc:333] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: stopping tablet replica
I20251027 03:10:26.212836 28253 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.212874 28253 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.213019 28253 tablet_replica.cc:333] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: stopping tablet replica
I20251027 03:10:26.213054 28253 raft_consensus.cc:2243] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.213073 28253 raft_consensus.cc:2272] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.213227 28253 tablet_replica.cc:333] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: stopping tablet replica
I20251027 03:10:26.213259 28253 raft_consensus.cc:2243] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.213289 28253 raft_consensus.cc:2272] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.213431 28253 tablet_replica.cc:333] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: stopping tablet replica
I20251027 03:10:26.213464 28253 raft_consensus.cc:2243] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:26.213492 28253 raft_consensus.cc:2272] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:26.224908 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:10:26.225878 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:26.227000 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:26.227169 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:26.227140 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:26.227160 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:26.227425 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:26.227487 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:26.227514 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534626227513 us; error 0 us; skew 500 ppm
I20251027 03:10:26.228080 28253 webserver.cc:492] Webserver started at http://127.27.151.68:38479/ using document root <none> and password file <none>
I20251027 03:10:26.228178 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:26.228225 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:26.228281 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:26.228569 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "4e4cd7564a494b93beec373c0a4c2775"
format_stamp: "Formatted at 2025-10-27 03:10:26 on dist-test-slave-4cc8"
I20251027 03:10:26.229473 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:26.229990 2126 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:26.230141 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:26.230188 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "4e4cd7564a494b93beec373c0a4c2775"
format_stamp: "Formatted at 2025-10-27 03:10:26 on dist-test-slave-4cc8"
I20251027 03:10:26.230238 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:26.236243 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:26.236395 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:26.236522 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:26.236690 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:26.236723 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:26.236758 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:26.236789 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:26.239200 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:45683
I20251027 03:10:26.241935 2189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:45683 every 8 connection(s)
I20251027 03:10:26.244580 2190 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:26.244647 2190 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:26.244711 2190 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:26.244911 1785 ts_manager.cc:194] Registered new tserver with Master: 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68:45683)
I20251027 03:10:26.245223 1785 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45528
W20251027 03:10:26.588732 1836 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:26.589421 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:26.589473 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:26.589967 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:26.590019 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:26.590378 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:26.590416 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
I20251027 03:10:27.245761 2190 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
W20251027 03:10:27.591221 1836 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:27.591872 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:27.591924 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:27.592365 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:27.592396 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:27.592809 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:27.592837 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:28.593678 1836 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:28.594293 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:28.594349 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:28.594707 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:28.594754 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:28.595106 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:28.595145 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:29.595502 1836 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table b38cbcc18f604c9aa65643c564b769eb: could not find any suitable replica to move from server d15b3c083d1441b79a054ba072903211 to server 4e4cd7564a494b93beec373c0a4c2775
W20251027 03:10:30.596542 1836 auto_rebalancer.cc:253] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:30.596939 1772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20251027 03:10:30.597303 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:30.597342 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:30.597708 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:30.597743 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.66:40215: connect: Connection refused (error 111)
W20251027 03:10:30.598091 1836 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:30.598124 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.27.151.65:35921: connect: Connection refused (error 111)
W20251027 03:10:31.598390 1836 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver d15b3c083d1441b79a054ba072903211 not available for placement
I20251027 03:10:31.753067 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:31.754261 2209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:31.754281 2206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.754317 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:31.754391 2207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.754611 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:31.754647 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:31.754662 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534631754661 us; error 0 us; skew 500 ppm
I20251027 03:10:31.755223 28253 webserver.cc:492] Webserver started at http://127.27.151.65:40825/ using document root <none> and password file <none>
I20251027 03:10:31.755301 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:31.755338 28253 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251027 03:10:31.755832 28253 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.756489 2214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.756598 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.756644 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "598c4e01f9244c27bb5c87ced4782621"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:31.756708 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:31.767786 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:31.767961 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:31.768077 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:31.768349 2221 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20251027 03:10:31.769506 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251027 03:10:31.769542 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:31.769573 28253 ts_tablet_manager.cc:600] Registering tablets (0/4 complete)
I20251027 03:10:31.770020 2221 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:31.770512 28253 ts_tablet_manager.cc:616] Registered 4 tablets
I20251027 03:10:31.770550 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:31.770900 2221 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.771061 2221 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap complete.
I20251027 03:10:31.771169 2221 ts_tablet_manager.cc:1403] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.771292 2221 raft_consensus.cc:359] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.771354 2221 raft_consensus.cc:740] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.771397 2221 consensus_queue.cc:260] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.771545 2221 ts_tablet_manager.cc:1434] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:31.771605 2221 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:31.772501 2221 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.772683 2221 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap complete.
I20251027 03:10:31.772770 2221 ts_tablet_manager.cc:1403] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:31.772876 2221 raft_consensus.cc:359] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.772933 2221 raft_consensus.cc:740] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.772976 2221 consensus_queue.cc:260] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.773064 2221 ts_tablet_manager.cc:1434] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:31.773114 2221 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
I20251027 03:10:31.773922 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:35921
I20251027 03:10:31.773968 2283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:35921 every 8 connection(s)
I20251027 03:10:31.774045 2221 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251027 03:10:31.774194 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:10:31.774267 2221 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap complete.
I20251027 03:10:31.774358 2221 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:31.774461 2221 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.774528 2221 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.774581 2221 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.774663 2221 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.774716 2221 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap starting.
W20251027 03:10:31.775154 2287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:31.775195 2291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.775553 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:31.775683 2221 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.775843 2221 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Bootstrap complete.
I20251027 03:10:31.775934 2221 ts_tablet_manager.cc:1403] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:31.776031 2221 raft_consensus.cc:359] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.776084 2221 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.776134 2221 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.776240 2221 ts_tablet_manager.cc:1434] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20251027 03:10:31.776464 2289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.776585 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:31.776620 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:31.776631 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534631776632 us; error 0 us; skew 500 ppm
I20251027 03:10:31.777235 28253 webserver.cc:492] Webserver started at http://127.27.151.66:38869/ using document root <none> and password file <none>
I20251027 03:10:31.777300 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:31.777334 28253 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251027 03:10:31.777767 28253 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.778292 2297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.778486 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.778523 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "d15b3c083d1441b79a054ba072903211"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:31.778563 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:31.781807 2284 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:31.781864 2284 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:31.781971 2284 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:31.782261 1785 ts_manager.cc:194] Re-registered known tserver with Master: 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921)
I20251027 03:10:31.782727 1785 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58986
I20251027 03:10:31.783066 2284 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
I20251027 03:10:31.800897 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:31.801076 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:31.801200 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:31.802512 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251027 03:10:31.802564 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:31.802969 2304 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:31.803463 28253 ts_tablet_manager.cc:616] Registered 4 tablets
I20251027 03:10:31.803498 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:31.803884 2304 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251027 03:10:31.804064 2304 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Bootstrap complete.
I20251027 03:10:31.804183 2304 ts_tablet_manager.cc:1403] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.001s
I20251027 03:10:31.804325 2304 raft_consensus.cc:359] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.804395 2304 raft_consensus.cc:740] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.804467 2304 consensus_queue.cc:260] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.804613 2304 ts_tablet_manager.cc:1434] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.804675 2304 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:31.805557 2304 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251027 03:10:31.805750 2304 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Bootstrap complete.
I20251027 03:10:31.805837 2304 ts_tablet_manager.cc:1403] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:31.805967 2304 raft_consensus.cc:359] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.806023 2304 raft_consensus.cc:740] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.806078 2304 consensus_queue.cc:260] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.806154 2304 ts_tablet_manager.cc:1434] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.806231 2304 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:31.806928 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:40215
I20251027 03:10:31.806977 2365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:40215 every 8 connection(s)
I20251027 03:10:31.807193 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:10:31.807209 2304 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.807384 2304 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Bootstrap complete.
I20251027 03:10:31.807471 2304 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:31.807600 2304 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.807641 2304 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.807701 2304 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.807789 2304 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.807838 2304 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Bootstrap starting.
I20251027 03:10:31.809019 2304 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20251027 03:10:31.809151 2370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.809233 2304 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Bootstrap complete.
W20251027 03:10:31.809320 2371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.809332 2304 ts_tablet_manager.cc:1403] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.809506 2304 raft_consensus.cc:359] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.809607 2304 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15b3c083d1441b79a054ba072903211, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.809764 2304 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.809989 2304 ts_tablet_manager.cc:1434] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20251027 03:10:31.810462 2373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:31.810479 2367 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:31.810537 2367 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:31.810596 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:31.810632 2367 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:31.810703 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:31.810729 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:31.810740 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534631810740 us; error 0 us; skew 500 ppm
I20251027 03:10:31.810933 1785 ts_manager.cc:194] Re-registered known tserver with Master: d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215)
I20251027 03:10:31.811251 28253 webserver.cc:492] Webserver started at http://127.27.151.67:39523/ using document root <none> and password file <none>
I20251027 03:10:31.811316 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:31.811348 28253 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251027 03:10:31.811365 1785 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58998
I20251027 03:10:31.811689 2367 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
I20251027 03:10:31.811836 28253 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.812348 2379 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.812449 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.812490 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "39262c5b16884c76b3674e96334e5d72"
format_stamp: "Formatted at 2025-10-27 03:10:24 on dist-test-slave-4cc8"
I20251027 03:10:31.812531 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:31.817193 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:31.817404 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:31.817541 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:31.818773 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251027 03:10:31.818805 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:31.819162 2386 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:31.819624 28253 ts_tablet_manager.cc:616] Registered 4 tablets
I20251027 03:10:31.819664 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:31.820137 2386 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.820353 2386 tablet_bootstrap.cc:492] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Bootstrap complete.
I20251027 03:10:31.820438 2386 ts_tablet_manager.cc:1403] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.820571 2386 raft_consensus.cc:359] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.820629 2386 raft_consensus.cc:740] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.820683 2386 consensus_queue.cc:260] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.820860 2386 ts_tablet_manager.cc:1434] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:31.820919 2386 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:31.821799 2386 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.821949 2386 tablet_bootstrap.cc:492] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Bootstrap complete.
I20251027 03:10:31.822041 2386 ts_tablet_manager.cc:1403] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.822173 2386 raft_consensus.cc:359] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.822216 2386 raft_consensus.cc:740] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.822280 2386 consensus_queue.cc:260] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.822377 2386 ts_tablet_manager.cc:1434] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:31.822430 2386 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:31.823302 2386 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.823463 2386 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Bootstrap complete.
I20251027 03:10:31.823508 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:41635
I20251027 03:10:31.823545 2386 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.823690 2386 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.823761 2386 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.823959 2386 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.824074 2386 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:31.824126 2386 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Bootstrap starting.
I20251027 03:10:31.824244 2447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:41635 every 8 connection(s)
I20251027 03:10:31.825193 2386 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:31.825362 2386 tablet_bootstrap.cc:492] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Bootstrap complete.
I20251027 03:10:31.825454 2386 ts_tablet_manager.cc:1403] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.825601 2386 raft_consensus.cc:359] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.825659 2386 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Initialized, Role: FOLLOWER
I20251027 03:10:31.825742 2386 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:31.826097 2386 ts_tablet_manager.cc:1434] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:31.826470 2448 heartbeater.cc:344] Connected to a master server at 127.27.151.126:36947
I20251027 03:10:31.826522 2448 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:31.826619 2448 heartbeater.cc:507] Master 127.27.151.126:36947 requested a full tablet report, sending...
I20251027 03:10:31.826823 1785 ts_manager.cc:194] Re-registered known tserver with Master: 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:31.827095 1785 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59004
I20251027 03:10:31.827381 2448 heartbeater.cc:499] Master 127.27.151.126:36947 was elected leader, sending a full tablet report...
W20251027 03:10:32.602154 1836 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Replica d15b3c083d1441b79a054ba072903211 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } }
W20251027 03:10:32.602680 1836 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a8553ab4d14b48d3a8da81572d5e2882, TS 39262c5b16884c76b3674e96334e5d72 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:32.602736 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet a8553ab4d14b48d3a8da81572d5e2882, TS 39262c5b16884c76b3674e96334e5d72 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:32.605278 1836 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0dc9f313b28b48d2ae0711f6ede99030, TS d15b3c083d1441b79a054ba072903211 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:32.605325 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 0dc9f313b28b48d2ae0711f6ede99030, TS d15b3c083d1441b79a054ba072903211 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:32.605648 1836 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 836f565450b04d548ab63bc15a22a254, TS 598c4e01f9244c27bb5c87ced4782621 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:32.605685 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 836f565450b04d548ab63bc15a22a254, TS 598c4e01f9244c27bb5c87ced4782621 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.063676 2455 raft_consensus.cc:493] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:33.063787 2455 raft_consensus.cc:515] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.063959 2455 leader_election.cc:290] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.066592 2340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211" is_pre_election: true
I20251027 03:10:33.066680 2340 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 1.
I20251027 03:10:33.066769 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:33.066820 2216 leader_election.cc:304] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.066854 2421 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 1.
I20251027 03:10:33.066926 2455 raft_consensus.cc:2804] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 03:10:33.066968 2455 raft_consensus.cc:493] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:33.066990 2455 raft_consensus.cc:3060] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.067559 2455 raft_consensus.cc:515] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.067667 2455 leader_election.cc:290] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 2 election: Requested vote from peers d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.067812 2340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211"
I20251027 03:10:33.067812 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dc9f313b28b48d2ae0711f6ede99030" candidate_uuid: "598c4e01f9244c27bb5c87ced4782621" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:33.067899 2340 raft_consensus.cc:3060] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.067917 2421 raft_consensus.cc:3060] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.068655 2421 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 2.
I20251027 03:10:33.068660 2340 raft_consensus.cc:2468] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 598c4e01f9244c27bb5c87ced4782621 in term 2.
I20251027 03:10:33.068826 2216 leader_election.cc:304] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.068912 2455 raft_consensus.cc:2804] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:10:33.069027 2455 raft_consensus.cc:697] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 LEADER]: Becoming Leader. State: Replica: 598c4e01f9244c27bb5c87ced4782621, State: Running, Role: LEADER
I20251027 03:10:33.069072 2455 consensus_queue.cc:237] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.069540 1785 catalog_manager.cc:5649] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "598c4e01f9244c27bb5c87ced4782621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:33.158138 2465 raft_consensus.cc:493] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:33.158221 2465 raft_consensus.cc:515] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.158411 2465 leader_election.cc:290] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.160946 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:33.161026 2257 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:33.160952 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:33.161080 2421 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:33.161245 2298 leader_election.cc:304] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.161327 2465 raft_consensus.cc:2804] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 03:10:33.161366 2465 raft_consensus.cc:493] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:33.161389 2465 raft_consensus.cc:3060] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.161988 2465 raft_consensus.cc:515] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.162083 2465 leader_election.cc:290] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.162243 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:33.162287 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "606568a1268245b3839b89f59239ef13" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:33.162295 2257 raft_consensus.cc:3060] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.162365 2421 raft_consensus.cc:3060] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.162925 2421 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 2.
I20251027 03:10:33.162930 2257 raft_consensus.cc:2468] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 2.
I20251027 03:10:33.163070 2298 leader_election.cc:304] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.163141 2465 raft_consensus.cc:2804] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:10:33.163242 2465 raft_consensus.cc:697] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 2 LEADER]: Becoming Leader. State: Replica: d15b3c083d1441b79a054ba072903211, State: Running, Role: LEADER
I20251027 03:10:33.163307 2465 consensus_queue.cc:237] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.163789 1785 catalog_manager.cc:5649] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d15b3c083d1441b79a054ba072903211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:33.421113 2465 raft_consensus.cc:493] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:33.421221 2465 raft_consensus.cc:515] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.421353 2465 leader_election.cc:290] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.421604 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:33.421604 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:33.421710 2257 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:33.421715 2421 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 1.
I20251027 03:10:33.421876 2298 leader_election.cc:304] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.421972 2465 raft_consensus.cc:2804] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 03:10:33.422012 2465 raft_consensus.cc:493] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:33.422034 2465 raft_consensus.cc:3060] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.422623 2465 raft_consensus.cc:515] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.422719 2465 leader_election.cc:290] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.422873 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:33.422911 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8553ab4d14b48d3a8da81572d5e2882" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:33.422963 2257 raft_consensus.cc:3060] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.422994 2421 raft_consensus.cc:3060] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.423491 2257 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 2.
I20251027 03:10:33.423502 2421 raft_consensus.cc:2468] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 2.
I20251027 03:10:33.423640 2298 leader_election.cc:304] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.423720 2465 raft_consensus.cc:2804] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:10:33.423769 2465 raft_consensus.cc:697] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 2 LEADER]: Becoming Leader. State: Replica: d15b3c083d1441b79a054ba072903211, State: Running, Role: LEADER
I20251027 03:10:33.423806 2465 consensus_queue.cc:237] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.424311 1785 catalog_manager.cc:5649] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d15b3c083d1441b79a054ba072903211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:33.499555 2421 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:10:33.499832 2455 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:33.501304 2340 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:10:33.501482 2476 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:33.566485 2478 raft_consensus.cc:493] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:33.566582 2478 raft_consensus.cc:515] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.566783 2478 leader_election.cc:290] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215)
I20251027 03:10:33.569831 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "39262c5b16884c76b3674e96334e5d72" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:33.569831 2340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "39262c5b16884c76b3674e96334e5d72" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211" is_pre_election: true
I20251027 03:10:33.569967 2257 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39262c5b16884c76b3674e96334e5d72 in term 1.
I20251027 03:10:33.569985 2340 raft_consensus.cc:2410] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39262c5b16884c76b3674e96334e5d72 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251027 03:10:33.570153 2380 leader_election.cc:304] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, 598c4e01f9244c27bb5c87ced4782621; no voters: d15b3c083d1441b79a054ba072903211
I20251027 03:10:33.570235 2478 raft_consensus.cc:2804] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 03:10:33.570274 2478 raft_consensus.cc:493] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:33.570293 2478 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.570894 2478 raft_consensus.cc:515] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.571002 2478 leader_election.cc:290] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [CANDIDATE]: Term 2 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215)
I20251027 03:10:33.571130 2340 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "39262c5b16884c76b3674e96334e5d72" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d15b3c083d1441b79a054ba072903211"
I20251027 03:10:33.571154 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "39262c5b16884c76b3674e96334e5d72" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:33.571210 2340 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.571223 2257 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 1 FOLLOWER]: Advancing to term 2
I20251027 03:10:33.571833 2340 raft_consensus.cc:2410] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39262c5b16884c76b3674e96334e5d72 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251027 03:10:33.571875 2257 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39262c5b16884c76b3674e96334e5d72 in term 2.
I20251027 03:10:33.571995 2380 leader_election.cc:304] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39262c5b16884c76b3674e96334e5d72, 598c4e01f9244c27bb5c87ced4782621; no voters:
I20251027 03:10:33.572077 2478 raft_consensus.cc:2804] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 03:10:33.572124 2478 raft_consensus.cc:697] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 2 LEADER]: Becoming Leader. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Running, Role: LEADER
I20251027 03:10:33.572186 2478 consensus_queue.cc:237] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.572691 1786 catalog_manager.cc:5649] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 reported cstate change: term changed from 1 to 2, leader changed from d15b3c083d1441b79a054ba072903211 (127.27.151.66) to 39262c5b16884c76b3674e96334e5d72 (127.27.151.67). New cstate: current_term: 2 leader_uuid: "39262c5b16884c76b3674e96334e5d72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:33.579447 2257 raft_consensus.cc:1275] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Refusing update from remote peer d15b3c083d1441b79a054ba072903211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:10:33.579617 2465 consensus_queue.cc:1048] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:33.580701 2421 raft_consensus.cc:1275] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Refusing update from remote peer d15b3c083d1441b79a054ba072903211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:10:33.580880 2486 consensus_queue.cc:1048] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:33.587178 2486 raft_consensus.cc:493] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:33.587229 2486 raft_consensus.cc:515] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.587301 2486 leader_election.cc:290] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.587436 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621" is_pre_election: true
I20251027 03:10:33.587483 2257 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 2.
I20251027 03:10:33.587477 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72" is_pre_election: true
I20251027 03:10:33.587591 2298 leader_election.cc:304] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters:
I20251027 03:10:33.587706 2486 raft_consensus.cc:2804] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251027 03:10:33.587738 2486 raft_consensus.cc:493] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:33.587750 2486 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:10:33.588349 2486 raft_consensus.cc:515] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.588505 2486 leader_election.cc:290] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 3 election: Requested vote from peers 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921), 39262c5b16884c76b3674e96334e5d72 (127.27.151.67:41635)
I20251027 03:10:33.588550 2257 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "598c4e01f9244c27bb5c87ced4782621"
I20251027 03:10:33.588615 2257 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:10:33.588617 2421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "836f565450b04d548ab63bc15a22a254" candidate_uuid: "d15b3c083d1441b79a054ba072903211" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39262c5b16884c76b3674e96334e5d72"
I20251027 03:10:33.589156 2257 raft_consensus.cc:2468] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d15b3c083d1441b79a054ba072903211 in term 3.
I20251027 03:10:33.589289 2298 leader_election.cc:304] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 598c4e01f9244c27bb5c87ced4782621, d15b3c083d1441b79a054ba072903211; no voters: 39262c5b16884c76b3674e96334e5d72
I20251027 03:10:33.589370 2486 raft_consensus.cc:2804] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 03:10:33.589416 2486 raft_consensus.cc:697] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 3 LEADER]: Becoming Leader. State: Replica: d15b3c083d1441b79a054ba072903211, State: Running, Role: LEADER
I20251027 03:10:33.589449 2486 consensus_queue.cc:237] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.589882 1785 catalog_manager.cc:5649] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 reported cstate change: term changed from 2 to 3, leader changed from 39262c5b16884c76b3674e96334e5d72 (127.27.151.67) to d15b3c083d1441b79a054ba072903211 (127.27.151.66). New cstate: current_term: 3 leader_uuid: "d15b3c083d1441b79a054ba072903211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:33.606216 2257 consensus_queue.cc:237] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } }
I20251027 03:10:33.606621 2340 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:10:33.606629 2421 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251027 03:10:33.606957 2455 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:33.607084 2459 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:33.607517 2459 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } } }
I20251027 03:10:33.607630 2340 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } } }
I20251027 03:10:33.608511 2421 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } } }
I20251027 03:10:33.608660 1785 catalog_manager.cc:5649] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 reported cstate change: config changed from index -1 to 3, NON_VOTER 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) added. New cstate: current_term: 2 leader_uuid: "598c4e01f9244c27bb5c87ced4782621" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251027 03:10:33.609560 1836 auto_rebalancer.cc:253] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20251027 03:10:33.610270 1836 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 836f565450b04d548ab63bc15a22a254, TS 39262c5b16884c76b3674e96334e5d72 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:33.610323 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 836f565450b04d548ab63bc15a22a254, TS 39262c5b16884c76b3674e96334e5d72 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:33.610352 2218 consensus_peers.cc:597] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 -> Peer 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68:45683): Couldn't send request to peer 4e4cd7564a494b93beec373c0a4c2775. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dc9f313b28b48d2ae0711f6ede99030. This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:33.610882 1836 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 606568a1268245b3839b89f59239ef13, TS 598c4e01f9244c27bb5c87ced4782621 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
W20251027 03:10:33.610920 1836 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 606568a1268245b3839b89f59239ef13, TS 598c4e01f9244c27bb5c87ced4782621 -> TS 4e4cd7564a494b93beec373c0a4c2775 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:33.822192 2421 raft_consensus.cc:1275] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Refusing update from remote peer d15b3c083d1441b79a054ba072903211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:10:33.822414 2486 consensus_queue.cc:1048] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:33.823628 2257 raft_consensus.cc:1275] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Refusing update from remote peer d15b3c083d1441b79a054ba072903211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251027 03:10:33.823868 2486 consensus_queue.cc:1048] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:34.000433 2493 ts_tablet_manager.cc:933] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: Initiating tablet copy from peer 598c4e01f9244c27bb5c87ced4782621 (127.27.151.65:35921)
I20251027 03:10:34.000625 2421 raft_consensus.cc:3055] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 2 LEADER]: Stepping down as leader of term 2
I20251027 03:10:34.000667 2493 tablet_copy_client.cc:323] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.65:35921
I20251027 03:10:34.000710 2421 raft_consensus.cc:740] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39262c5b16884c76b3674e96334e5d72, State: Running, Role: LEADER
I20251027 03:10:34.000766 2421 consensus_queue.cc:260] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } }
I20251027 03:10:34.000854 2421 raft_consensus.cc:3060] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Advancing to term 3
I20251027 03:10:34.001679 2421 raft_consensus.cc:1275] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 3 FOLLOWER]: Refusing update from remote peer d15b3c083d1441b79a054ba072903211: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251027 03:10:34.002218 2486 consensus_queue.cc:1048] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:34.002354 2486 consensus_queue.cc:1243] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [LEADER]: Peer 39262c5b16884c76b3674e96334e5d72 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251027 03:10:34.002709 2421 pending_rounds.cc:85] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Aborting all ops after (but not including) 0
I20251027 03:10:34.002776 2421 pending_rounds.cc:107] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20251027 03:10:34.002796 2421 raft_consensus.cc:2889] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251027 03:10:34.003638 2257 raft_consensus.cc:1275] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 3 FOLLOWER]: Refusing update from remote peer d15b3c083d1441b79a054ba072903211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251027 03:10:34.003974 2261 tablet_copy_service.cc:140] P 598c4e01f9244c27bb5c87ced4782621: Received BeginTabletCopySession request for tablet 0dc9f313b28b48d2ae0711f6ede99030 from peer 4e4cd7564a494b93beec373c0a4c2775 ({username='slave'} at 127.0.0.1:39134)
I20251027 03:10:34.004030 2261 tablet_copy_service.cc:161] P 598c4e01f9244c27bb5c87ced4782621: Beginning new tablet copy session on tablet 0dc9f313b28b48d2ae0711f6ede99030 from peer 4e4cd7564a494b93beec373c0a4c2775 at {username='slave'} at 127.0.0.1:39134: session id = 4e4cd7564a494b93beec373c0a4c2775-0dc9f313b28b48d2ae0711f6ede99030
I20251027 03:10:34.004033 2465 consensus_queue.cc:1048] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:34.004696 2261 tablet_copy_source_session.cc:215] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:34.005007 2493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dc9f313b28b48d2ae0711f6ede99030. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:34.005916 2493 tablet_copy_client.cc:806] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:34.006039 2493 tablet_copy_client.cc:670] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:34.006474 2493 tablet_copy_client.cc:538] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:34.007197 2493 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: Bootstrap starting.
I20251027 03:10:34.008543 2493 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:34.008718 2493 tablet_bootstrap.cc:492] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: Bootstrap complete.
I20251027 03:10:34.008822 2493 ts_tablet_manager.cc:1403] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251027 03:10:34.008968 2493 raft_consensus.cc:359] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } }
I20251027 03:10:34.009043 2493 raft_consensus.cc:740] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4e4cd7564a494b93beec373c0a4c2775, State: Initialized, Role: LEARNER
I20251027 03:10:34.009079 2493 consensus_queue.cc:260] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: true } }
I20251027 03:10:34.009220 2493 ts_tablet_manager.cc:1434] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:34.009384 2261 tablet_copy_service.cc:342] P 598c4e01f9244c27bb5c87ced4782621: Request end of tablet copy session 4e4cd7564a494b93beec373c0a4c2775-0dc9f313b28b48d2ae0711f6ede99030 received from {username='slave'} at 127.0.0.1:39134
I20251027 03:10:34.009425 2261 tablet_copy_service.cc:434] P 598c4e01f9244c27bb5c87ced4782621: ending tablet copy session 4e4cd7564a494b93beec373c0a4c2775-0dc9f313b28b48d2ae0711f6ede99030 on tablet 0dc9f313b28b48d2ae0711f6ede99030 with peer 4e4cd7564a494b93beec373c0a4c2775
I20251027 03:10:34.408649 2164 raft_consensus.cc:1217] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251027 03:10:35.013671 2500 raft_consensus.cc:1064] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621: attempting to promote NON_VOTER 4e4cd7564a494b93beec373c0a4c2775 to VOTER
I20251027 03:10:35.013890 2500 consensus_queue.cc:237] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } }
I20251027 03:10:35.014341 2164 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 LEARNER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251027 03:10:35.014371 2421 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251027 03:10:35.014384 2340 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251027 03:10:35.014531 2503 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:35.014637 2502 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:35.014740 2499 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:35.015388 2499 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.015491 2421 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.015852 2164 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.016686 2340 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.016764 1785 catalog_manager.cc:5649] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 reported cstate change: config changed from index 3 to 4, 4e4cd7564a494b93beec373c0a4c2775 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "598c4e01f9244c27bb5c87ced4782621" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d15b3c083d1441b79a054ba072903211" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 40215 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251027 03:10:35.018456 2257 consensus_queue.cc:237] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } }
I20251027 03:10:35.018788 2164 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251027 03:10:35.018795 2421 raft_consensus.cc:1275] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Refusing update from remote peer 598c4e01f9244c27bb5c87ced4782621: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251027 03:10:35.018952 2503 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 03:10:35.019032 2502 consensus_queue.cc:1048] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251027 03:10:35.019554 2500 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d15b3c083d1441b79a054ba072903211 (127.27.151.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.019649 2164 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d15b3c083d1441b79a054ba072903211 (127.27.151.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.020372 1830 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0dc9f313b28b48d2ae0711f6ede99030 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:35.020473 1787 catalog_manager.cc:5649] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 reported cstate change: config changed from index 4 to 5, VOTER d15b3c083d1441b79a054ba072903211 (127.27.151.66) evicted. New cstate: current_term: 2 leader_uuid: "598c4e01f9244c27bb5c87ced4782621" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.020474 2421 raft_consensus.cc:2955] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER d15b3c083d1441b79a054ba072903211 (127.27.151.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "598c4e01f9244c27bb5c87ced4782621" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 35921 } } peers { permanent_uuid: "39262c5b16884c76b3674e96334e5d72" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 41635 } } peers { permanent_uuid: "4e4cd7564a494b93beec373c0a4c2775" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 45683 } attrs { promote: false } } }
I20251027 03:10:35.023438 2328 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dc9f313b28b48d2ae0711f6ede99030 with delete_type TABLET_DATA_TOMBSTONED (TS d15b3c083d1441b79a054ba072903211 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:37348
I20251027 03:10:35.023612 2515 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.023665 2515 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.023701 2515 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.023900 2515 ts_tablet_manager.cc:1916] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:10:35.024930 2515 ts_tablet_manager.cc:1929] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20251027 03:10:35.024976 2515 log.cc:1199] T 0dc9f313b28b48d2ae0711f6ede99030 P d15b3c083d1441b79a054ba072903211: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/0dc9f313b28b48d2ae0711f6ede99030
I20251027 03:10:35.025262 1772 catalog_manager.cc:4985] TS d15b3c083d1441b79a054ba072903211 (127.27.151.66:40215): tablet 0dc9f313b28b48d2ae0711f6ede99030 (table test-workload [id=b38cbcc18f604c9aa65643c564b769eb]) successfully deleted
I20251027 03:10:35.848364 28253 tablet_server.cc:178] TabletServer@127.27.151.65:35921 shutting down...
I20251027 03:10:35.850544 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:35.850659 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.850728 28253 raft_consensus.cc:2243] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.850776 28253 raft_consensus.cc:2272] T 836f565450b04d548ab63bc15a22a254 P 598c4e01f9244c27bb5c87ced4782621 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.850941 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.850975 28253 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:10:35.851028 28253 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.851174 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.851205 28253 raft_consensus.cc:2243] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.851240 28253 raft_consensus.cc:2272] T 606568a1268245b3839b89f59239ef13 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.851346 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.851372 28253 raft_consensus.cc:2243] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.851405 28253 raft_consensus.cc:2272] T a8553ab4d14b48d3a8da81572d5e2882 P 598c4e01f9244c27bb5c87ced4782621 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.863389 28253 tablet_server.cc:195] TabletServer@127.27.151.65:35921 shutdown complete.
I20251027 03:10:35.864400 28253 tablet_server.cc:178] TabletServer@127.27.151.66:40215 shutting down...
I20251027 03:10:35.866367 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:35.866451 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.866503 28253 raft_consensus.cc:2243] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 3 LEADER]: Raft consensus shutting down.
I20251027 03:10:35.866565 28253 raft_consensus.cc:2272] T 836f565450b04d548ab63bc15a22a254 P d15b3c083d1441b79a054ba072903211 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.866711 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.866747 28253 raft_consensus.cc:2243] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:10:35.866798 28253 raft_consensus.cc:2272] T 606568a1268245b3839b89f59239ef13 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.866904 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.866937 28253 raft_consensus.cc:2243] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 2 LEADER]: Raft consensus shutting down.
I20251027 03:10:35.866984 28253 raft_consensus.cc:2272] T a8553ab4d14b48d3a8da81572d5e2882 P d15b3c083d1441b79a054ba072903211 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.878631 28253 tablet_server.cc:195] TabletServer@127.27.151.66:40215 shutdown complete.
I20251027 03:10:35.879468 28253 tablet_server.cc:178] TabletServer@127.27.151.67:41635 shutting down...
I20251027 03:10:35.882112 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:35.882202 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.882261 28253 raft_consensus.cc:2243] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.882299 28253 raft_consensus.cc:2272] T 836f565450b04d548ab63bc15a22a254 P 39262c5b16884c76b3674e96334e5d72 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.882429 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.882458 28253 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.882494 28253 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.882648 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.882684 28253 raft_consensus.cc:2243] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.882706 28253 raft_consensus.cc:2272] T 606568a1268245b3839b89f59239ef13 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.882823 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:35.882848 28253 raft_consensus.cc:2243] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.882876 28253 raft_consensus.cc:2272] T a8553ab4d14b48d3a8da81572d5e2882 P 39262c5b16884c76b3674e96334e5d72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.894076 28253 tablet_server.cc:195] TabletServer@127.27.151.67:41635 shutdown complete.
I20251027 03:10:35.894897 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:10:35.897202 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:35.897293 28253 tablet_replica.cc:333] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775: stopping tablet replica
I20251027 03:10:35.897343 28253 raft_consensus.cc:2243] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:35.897395 28253 raft_consensus.cc:2272] T 0dc9f313b28b48d2ae0711f6ede99030 P 4e4cd7564a494b93beec373c0a4c2775 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.908635 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:10:35.909332 28253 master.cc:561] Master@127.27.151.126:36947 shutting down...
I20251027 03:10:35.911715 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:35.911788 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:35.911830 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8f5561234ce244118f40ffc2d62d87b7: stopping tablet replica
I20251027 03:10:35.923153 28253 master.cc:583] Master@127.27.151.126:36947 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11361 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20251027 03:10:35.928488 28253 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.151.126:35495
I20251027 03:10:35.928630 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:35.929654 2517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:35.929634 2516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:35.929733 2519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:35.929774 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:35.929951 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:35.929986 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:35.930007 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534635930008 us; error 0 us; skew 500 ppm
I20251027 03:10:35.930601 28253 webserver.cc:492] Webserver started at http://127.27.151.126:37433/ using document root <none> and password file <none>
I20251027 03:10:35.930709 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:35.930753 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:35.930802 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:35.931052 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/master-0-root/instance:
uuid: "7e09f0d52e24423897dc971fcbafefd4"
format_stamp: "Formatted at 2025-10-27 03:10:35 on dist-test-slave-4cc8"
I20251027 03:10:35.931843 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:35.932263 2524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:35.932391 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:35.932433 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/master-0-root
uuid: "7e09f0d52e24423897dc971fcbafefd4"
format_stamp: "Formatted at 2025-10-27 03:10:35 on dist-test-slave-4cc8"
I20251027 03:10:35.932477 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:35.939152 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:35.939302 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:35.941673 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.126:35495
I20251027 03:10:35.944522 2576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.126:35495 every 8 connection(s)
I20251027 03:10:35.944624 2577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:35.945538 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Bootstrap starting.
I20251027 03:10:35.945967 2577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:35.946492 2577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: No bootstrap required, opened a new log
I20251027 03:10:35.946645 2577 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e09f0d52e24423897dc971fcbafefd4" member_type: VOTER }
I20251027 03:10:35.946694 2577 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:35.946719 2577 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e09f0d52e24423897dc971fcbafefd4, State: Initialized, Role: FOLLOWER
I20251027 03:10:35.946781 2577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e09f0d52e24423897dc971fcbafefd4" member_type: VOTER }
I20251027 03:10:35.946815 2577 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 03:10:35.946837 2577 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 03:10:35.946866 2577 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:35.947332 2577 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e09f0d52e24423897dc971fcbafefd4" member_type: VOTER }
I20251027 03:10:35.947383 2577 leader_election.cc:304] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e09f0d52e24423897dc971fcbafefd4; no voters:
I20251027 03:10:35.947480 2577 leader_election.cc:290] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 03:10:35.947541 2580 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:35.947639 2577 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 03:10:35.947712 2580 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e09f0d52e24423897dc971fcbafefd4, State: Running, Role: LEADER
I20251027 03:10:35.947772 2580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e09f0d52e24423897dc971fcbafefd4" member_type: VOTER }
I20251027 03:10:35.947945 2581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e09f0d52e24423897dc971fcbafefd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e09f0d52e24423897dc971fcbafefd4" member_type: VOTER } }
I20251027 03:10:35.947959 2582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e09f0d52e24423897dc971fcbafefd4. Latest consensus state: current_term: 1 leader_uuid: "7e09f0d52e24423897dc971fcbafefd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e09f0d52e24423897dc971fcbafefd4" member_type: VOTER } }
I20251027 03:10:35.947996 2581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [sys.catalog]: This master's current role is: LEADER
I20251027 03:10:35.948041 2582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [sys.catalog]: This master's current role is: LEADER
I20251027 03:10:35.948223 2586 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 03:10:35.948348 2586 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 03:10:35.949070 28253 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251027 03:10:35.949462 2586 catalog_manager.cc:1357] Generated new cluster ID: ec853b9ad1764768bd497815422d905a
I20251027 03:10:35.949509 2586 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 03:10:35.958000 2586 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 03:10:35.958365 2586 catalog_manager.cc:1514] Loading token signing keys...
I20251027 03:10:35.965312 2586 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Generated new TSK 0
I20251027 03:10:35.965379 2586 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 03:10:35.980742 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:35.981930 2598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:35.981937 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:35.982033 2601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:35.982072 2599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:35.982316 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:35.982355 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:35.982369 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534635982369 us; error 0 us; skew 500 ppm
I20251027 03:10:35.982882 28253 webserver.cc:492] Webserver started at http://127.27.151.65:40263/ using document root <none> and password file <none>
I20251027 03:10:35.982967 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:35.983001 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:35.983035 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:35.983239 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/instance:
uuid: "0e07265804fe4b3a9c8214c7df139f1d"
format_stamp: "Formatted at 2025-10-27 03:10:35 on dist-test-slave-4cc8"
I20251027 03:10:35.983980 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:35.984390 2607 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:35.984495 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:35.984530 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "0e07265804fe4b3a9c8214c7df139f1d"
format_stamp: "Formatted at 2025-10-27 03:10:35 on dist-test-slave-4cc8"
I20251027 03:10:35.984555 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:35.998878 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:35.999029 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:35.999130 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:35.999258 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:35.999280 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:35.999300 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:35.999311 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.002110 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:37617
I20251027 03:10:36.002144 2670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:37617 every 8 connection(s)
I20251027 03:10:36.002434 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:36.003412 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:36.003432 2678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:36.003489 2676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:36.003825 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:36.003943 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:36.003973 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:36.003989 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534636003989 us; error 0 us; skew 500 ppm
I20251027 03:10:36.004549 28253 webserver.cc:492] Webserver started at http://127.27.151.66:43547/ using document root <none> and password file <none>
I20251027 03:10:36.004616 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:36.004671 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:36.004722 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:36.004995 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/instance:
uuid: "b7f03efa0cee4eecbba43f88bd2724a7"
format_stamp: "Formatted at 2025-10-27 03:10:36 on dist-test-slave-4cc8"
I20251027 03:10:36.005827 2671 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35495
I20251027 03:10:36.005885 2671 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:36.005919 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:36.006012 2671 heartbeater.cc:507] Master 127.27.151.126:35495 requested a full tablet report, sending...
I20251027 03:10:36.006250 2541 ts_manager.cc:194] Registered new tserver with Master: 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617)
I20251027 03:10:36.006408 2683 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.006516 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.006562 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root
uuid: "b7f03efa0cee4eecbba43f88bd2724a7"
format_stamp: "Formatted at 2025-10-27 03:10:36 on dist-test-slave-4cc8"
I20251027 03:10:36.006605 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:36.006966 2541 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38934
I20251027 03:10:36.011072 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:36.011255 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:36.011369 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:36.011525 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:36.011552 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.011581 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:36.011606 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.014182 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.66:46409
I20251027 03:10:36.014238 2746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.66:46409 every 8 connection(s)
I20251027 03:10:36.014509 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 03:10:36.015422 2750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:36.015466 2751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:36.015722 2753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:36.016119 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:36.016253 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:36.016287 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:36.016305 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534636016305 us; error 0 us; skew 500 ppm
I20251027 03:10:36.016877 28253 webserver.cc:492] Webserver started at http://127.27.151.67:34291/ using document root <none> and password file <none>
I20251027 03:10:36.016961 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:36.017009 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:36.017056 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:36.017326 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root/instance:
uuid: "85419970b418485aa1e22a2bd96db12f"
format_stamp: "Formatted at 2025-10-27 03:10:36 on dist-test-slave-4cc8"
I20251027 03:10:36.017490 2747 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35495
I20251027 03:10:36.017544 2747 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:36.017630 2747 heartbeater.cc:507] Master 127.27.151.126:35495 requested a full tablet report, sending...
I20251027 03:10:36.017885 2541 ts_manager.cc:194] Registered new tserver with Master: b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:36.018326 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:36.018308 2541 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38940
I20251027 03:10:36.018847 2758 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.018965 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.019018 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root
uuid: "85419970b418485aa1e22a2bd96db12f"
format_stamp: "Formatted at 2025-10-27 03:10:36 on dist-test-slave-4cc8"
I20251027 03:10:36.019058 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:36.024374 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:36.024507 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:36.024598 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:36.024740 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:36.024762 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.024786 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:36.024803 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:36.027488 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.67:45727
I20251027 03:10:36.027539 2823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.67:45727 every 8 connection(s)
I20251027 03:10:36.029971 2824 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35495
I20251027 03:10:36.030019 2824 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:36.030102 2824 heartbeater.cc:507] Master 127.27.151.126:35495 requested a full tablet report, sending...
I20251027 03:10:36.030287 2541 ts_manager.cc:194] Registered new tserver with Master: 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:36.030596 2541 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38956
I20251027 03:10:36.030941 28253 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003267473s
I20251027 03:10:37.007480 2671 heartbeater.cc:499] Master 127.27.151.126:35495 was elected leader, sending a full tablet report...
I20251027 03:10:37.018889 2747 heartbeater.cc:499] Master 127.27.151.126:35495 was elected leader, sending a full tablet report...
I20251027 03:10:37.031005 2824 heartbeater.cc:499] Master 127.27.151.126:35495 was elected leader, sending a full tablet report...
I20251027 03:10:37.056012 28253 test_util.cc:276] Using random seed: 1045241866
I20251027 03:10:37.063809 2541 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38958:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:10:37.063953 2541 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:10:37.068531 2785 tablet_service.cc:1505] Processing CreateTablet for tablet 201f9e8ee9304d1585da90d7f2f0d272 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:37.068526 2786 tablet_service.cc:1505] Processing CreateTablet for tablet 6d7622b5f498448088e0344b369cdaa4 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:37.068522 2787 tablet_service.cc:1505] Processing CreateTablet for tablet f5293919b0184807b8b08ab4ba8bd2ec (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:37.068733 2784 tablet_service.cc:1505] Processing CreateTablet for tablet bd9a239c4fa148f981838fd3d0884c50 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:37.068672 2785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201f9e8ee9304d1585da90d7f2f0d272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.068814 2784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a239c4fa148f981838fd3d0884c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.068858 2787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5293919b0184807b8b08ab4ba8bd2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.069092 2711 tablet_service.cc:1505] Processing CreateTablet for tablet f5293919b0184807b8b08ab4ba8bd2ec (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:37.069183 2711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5293919b0184807b8b08ab4ba8bd2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.070190 2786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7622b5f498448088e0344b369cdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.070226 2844 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.070298 2710 tablet_service.cc:1505] Processing CreateTablet for tablet 6d7622b5f498448088e0344b369cdaa4 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:37.070405 2845 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.070477 2709 tablet_service.cc:1505] Processing CreateTablet for tablet 201f9e8ee9304d1585da90d7f2f0d272 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:37.070622 2708 tablet_service.cc:1505] Processing CreateTablet for tablet bd9a239c4fa148f981838fd3d0884c50 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:37.070619 2844 tablet_bootstrap.cc:654] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.070416 2710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7622b5f498448088e0344b369cdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.071007 2709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201f9e8ee9304d1585da90d7f2f0d272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.071908 2708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a239c4fa148f981838fd3d0884c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.072662 2845 tablet_bootstrap.cc:654] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.072751 2634 tablet_service.cc:1505] Processing CreateTablet for tablet 6d7622b5f498448088e0344b369cdaa4 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:37.072845 2634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7622b5f498448088e0344b369cdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.073184 2635 tablet_service.cc:1505] Processing CreateTablet for tablet f5293919b0184807b8b08ab4ba8bd2ec (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:37.073284 2635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5293919b0184807b8b08ab4ba8bd2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.073346 2845 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.073392 2845 ts_tablet_manager.cc:1403] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:10:37.073539 2845 raft_consensus.cc:359] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.073594 2845 raft_consensus.cc:385] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.073612 2845 raft_consensus.cc:740] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.073676 2845 consensus_queue.cc:260] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.073819 2845 ts_tablet_manager.cc:1434] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:37.073982 2845 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.074285 2849 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.074395 2845 tablet_bootstrap.cc:654] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.074607 2633 tablet_service.cc:1505] Processing CreateTablet for tablet 201f9e8ee9304d1585da90d7f2f0d272 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:37.074702 2849 tablet_bootstrap.cc:654] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.074712 2633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201f9e8ee9304d1585da90d7f2f0d272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.074754 2632 tablet_service.cc:1505] Processing CreateTablet for tablet bd9a239c4fa148f981838fd3d0884c50 (DEFAULT_TABLE table=test-workload [id=164795683cff4e139e24f3460173cb15]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:37.074824 2632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a239c4fa148f981838fd3d0884c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.075237 2849 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.075273 2849 ts_tablet_manager.cc:1403] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.075397 2849 raft_consensus.cc:359] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.075441 2849 raft_consensus.cc:385] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.075457 2849 raft_consensus.cc:740] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.075491 2849 consensus_queue.cc:260] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.075623 2849 ts_tablet_manager.cc:1434] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.075778 2849 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.076191 2849 tablet_bootstrap.cc:654] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.076705 2849 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.076748 2849 ts_tablet_manager.cc:1403] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.076891 2849 raft_consensus.cc:359] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.076946 2849 raft_consensus.cc:385] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.076970 2849 raft_consensus.cc:740] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.077011 2849 consensus_queue.cc:260] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.077091 2849 ts_tablet_manager.cc:1434] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.077153 2849 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.077291 2845 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.077328 2845 ts_tablet_manager.cc:1403] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:10:37.077378 2844 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.077410 2844 ts_tablet_manager.cc:1403] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20251027 03:10:37.077466 2845 raft_consensus.cc:359] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.077544 2849 tablet_bootstrap.cc:654] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.077533 2845 raft_consensus.cc:385] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.077527 2844 raft_consensus.cc:359] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.077628 2845 raft_consensus.cc:740] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.077636 2844 raft_consensus.cc:385] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.077697 2844 raft_consensus.cc:740] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.077718 2845 consensus_queue.cc:260] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.077770 2844 consensus_queue.cc:260] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.077831 2845 ts_tablet_manager.cc:1434] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:37.077898 2845 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.077941 2844 ts_tablet_manager.cc:1434] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.078006 2844 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.078312 2845 tablet_bootstrap.cc:654] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.078361 2849 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.078406 2849 ts_tablet_manager.cc:1403] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.078542 2849 raft_consensus.cc:359] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.078603 2849 raft_consensus.cc:385] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.078626 2849 raft_consensus.cc:740] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.078672 2849 consensus_queue.cc:260] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.078753 2849 ts_tablet_manager.cc:1434] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.078799 2849 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.079054 2845 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.079102 2845 ts_tablet_manager.cc:1403] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.079209 2849 tablet_bootstrap.cc:654] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.079221 2845 raft_consensus.cc:359] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.079317 2845 raft_consensus.cc:385] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.079348 2845 raft_consensus.cc:740] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.079398 2845 consensus_queue.cc:260] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.079490 2845 ts_tablet_manager.cc:1434] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.078411 2844 tablet_bootstrap.cc:654] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.079550 2845 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.079957 2845 tablet_bootstrap.cc:654] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.080220 2849 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.080289 2849 ts_tablet_manager.cc:1403] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.080441 2849 raft_consensus.cc:359] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.080510 2849 raft_consensus.cc:385] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.080529 2849 raft_consensus.cc:740] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.080574 2849 consensus_queue.cc:260] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.080601 2845 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.080680 2845 ts_tablet_manager.cc:1403] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.080698 2849 ts_tablet_manager.cc:1434] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:37.080852 2845 raft_consensus.cc:359] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.080921 2845 raft_consensus.cc:385] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.080940 2845 raft_consensus.cc:740] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.080971 2844 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.080982 2845 consensus_queue.cc:260] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.081027 2844 ts_tablet_manager.cc:1403] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251027 03:10:37.081106 2845 ts_tablet_manager.cc:1434] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.081383 2844 raft_consensus.cc:359] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.081486 2844 raft_consensus.cc:385] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.081521 2844 raft_consensus.cc:740] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.081571 2844 consensus_queue.cc:260] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.081668 2844 ts_tablet_manager.cc:1434] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:37.081722 2844 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.082132 2844 tablet_bootstrap.cc:654] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.082666 2844 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.082716 2844 ts_tablet_manager.cc:1403] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.082841 2844 raft_consensus.cc:359] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.082891 2844 raft_consensus.cc:385] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.082922 2844 raft_consensus.cc:740] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.082962 2844 consensus_queue.cc:260] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.083036 2844 ts_tablet_manager.cc:1434] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.083086 2844 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.083459 2844 tablet_bootstrap.cc:654] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.083978 2844 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.084020 2844 ts_tablet_manager.cc:1403] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.084192 2844 raft_consensus.cc:359] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.084265 2844 raft_consensus.cc:385] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.084288 2844 raft_consensus.cc:740] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.084338 2844 consensus_queue.cc:260] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.084435 2844 ts_tablet_manager.cc:1434] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.149188 2848 raft_consensus.cc:493] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.149262 2848 raft_consensus.cc:515] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.149461 2848 leader_election.cc:290] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.152890 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d7622b5f498448088e0344b369cdaa4" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f" is_pre_election: true
I20251027 03:10:37.152890 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d7622b5f498448088e0344b369cdaa4" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.152989 2797 raft_consensus.cc:2468] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.152997 2645 raft_consensus.cc:2468] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.153110 2686 leader_election.cc:304] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85419970b418485aa1e22a2bd96db12f, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.153203 2848 raft_consensus.cc:2804] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.153250 2848 raft_consensus.cc:493] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.153276 2848 raft_consensus.cc:3060] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.153861 2848 raft_consensus.cc:515] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.154012 2848 leader_election.cc:290] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.154150 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d7622b5f498448088e0344b369cdaa4" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.154206 2645 raft_consensus.cc:3060] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.154692 2645 raft_consensus.cc:2468] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.154775 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d7622b5f498448088e0344b369cdaa4" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f"
I20251027 03:10:37.154824 2797 raft_consensus.cc:3060] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.154865 2687 leader_election.cc:304] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.154953 2848 raft_consensus.cc:2804] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.155077 2848 raft_consensus.cc:697] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Becoming Leader. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Running, Role: LEADER
I20251027 03:10:37.155135 2848 consensus_queue.cc:237] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.155390 2797 raft_consensus.cc:2468] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.155589 2540 catalog_manager.cc:5649] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:37.174867 2855 raft_consensus.cc:493] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.174939 2855 raft_consensus.cc:515] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.175092 2855 leader_election.cc:290] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:37.177593 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd9a239c4fa148f981838fd3d0884c50" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.177647 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd9a239c4fa148f981838fd3d0884c50" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" is_pre_election: true
I20251027 03:10:37.177654 2645 raft_consensus.cc:2468] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 0.
I20251027 03:10:37.177736 2721 raft_consensus.cc:2468] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 0.
I20251027 03:10:37.177848 2762 leader_election.cc:304] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.177934 2855 raft_consensus.cc:2804] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.178119 2855 raft_consensus.cc:493] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.178153 2855 raft_consensus.cc:3060] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.178650 2855 raft_consensus.cc:515] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.178754 2855 leader_election.cc:290] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:37.178889 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd9a239c4fa148f981838fd3d0884c50" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.178891 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bd9a239c4fa148f981838fd3d0884c50" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7"
I20251027 03:10:37.179030 2645 raft_consensus.cc:3060] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.179044 2721 raft_consensus.cc:3060] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.179256 2848 raft_consensus.cc:493] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.179301 2848 raft_consensus.cc:515] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.179394 2848 leader_election.cc:290] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.179564 2644 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "201f9e8ee9304d1585da90d7f2f0d272" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.179625 2644 raft_consensus.cc:2468] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.179649 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "201f9e8ee9304d1585da90d7f2f0d272" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f" is_pre_election: true
I20251027 03:10:37.179700 2645 raft_consensus.cc:2468] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 1.
I20251027 03:10:37.179713 2797 raft_consensus.cc:2468] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.179782 2721 raft_consensus.cc:2468] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 1.
I20251027 03:10:37.179831 2687 leader_election.cc:304] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.179926 2762 leader_election.cc:304] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.179997 2848 raft_consensus.cc:2804] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.180006 2855 raft_consensus.cc:2804] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.180040 2848 raft_consensus.cc:493] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.180101 2848 raft_consensus.cc:3060] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.180191 2855 raft_consensus.cc:697] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Becoming Leader. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Running, Role: LEADER
I20251027 03:10:37.180245 2855 consensus_queue.cc:237] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.180678 2848 raft_consensus.cc:515] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.180711 2540 catalog_manager.cc:5649] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f reported cstate change: term changed from 0 to 1, leader changed from <none> to 85419970b418485aa1e22a2bd96db12f (127.27.151.67). New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:37.180784 2848 leader_election.cc:290] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.180950 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "201f9e8ee9304d1585da90d7f2f0d272" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f"
I20251027 03:10:37.181005 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "201f9e8ee9304d1585da90d7f2f0d272" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.181008 2797 raft_consensus.cc:3060] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.181070 2645 raft_consensus.cc:3060] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.181528 2797 raft_consensus.cc:2468] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.181532 2645 raft_consensus.cc:2468] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.181685 2686 leader_election.cc:304] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85419970b418485aa1e22a2bd96db12f, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.181787 2848 raft_consensus.cc:2804] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.181839 2848 raft_consensus.cc:697] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Becoming Leader. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Running, Role: LEADER
I20251027 03:10:37.181885 2848 consensus_queue.cc:237] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.182308 2540 catalog_manager.cc:5649] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:37.281045 2855 raft_consensus.cc:493] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.281165 2855 raft_consensus.cc:515] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.281324 2855 leader_election.cc:290] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:37.281548 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5293919b0184807b8b08ab4ba8bd2ec" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" is_pre_election: true
I20251027 03:10:37.281548 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5293919b0184807b8b08ab4ba8bd2ec" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.281630 2721 raft_consensus.cc:2468] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 0.
I20251027 03:10:37.281682 2645 raft_consensus.cc:2468] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 0.
I20251027 03:10:37.281818 2762 leader_election.cc:304] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.281921 2855 raft_consensus.cc:2804] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.281963 2855 raft_consensus.cc:493] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.281977 2855 raft_consensus.cc:3060] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.282550 2855 raft_consensus.cc:515] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.282636 2855 leader_election.cc:290] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:37.282810 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5293919b0184807b8b08ab4ba8bd2ec" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.282850 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f5293919b0184807b8b08ab4ba8bd2ec" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7"
I20251027 03:10:37.282866 2645 raft_consensus.cc:3060] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.282919 2721 raft_consensus.cc:3060] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.283440 2721 raft_consensus.cc:2468] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 1.
I20251027 03:10:37.283458 2645 raft_consensus.cc:2468] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 1.
I20251027 03:10:37.283608 2762 leader_election.cc:304] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.283692 2855 raft_consensus.cc:2804] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.283741 2855 raft_consensus.cc:697] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Becoming Leader. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Running, Role: LEADER
I20251027 03:10:37.283797 2855 consensus_queue.cc:237] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.284365 2540 catalog_manager.cc:5649] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f reported cstate change: term changed from 0 to 1, leader changed from <none> to 85419970b418485aa1e22a2bd96db12f (127.27.151.67). New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:37.289299 28253 test_util.cc:276] Using random seed: 1045475153
I20251027 03:10:37.293655 2540 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38972:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 03:10:37.293787 2540 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 03:10:37.294991 2632 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea846b9e34942c0a73ed0f05266a9f9 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:37.295047 2784 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea846b9e34942c0a73ed0f05266a9f9 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:37.295102 2632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea846b9e34942c0a73ed0f05266a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.295130 2708 tablet_service.cc:1505] Processing CreateTablet for tablet 3ea846b9e34942c0a73ed0f05266a9f9 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251027 03:10:37.295156 2784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea846b9e34942c0a73ed0f05266a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.295192 2786 tablet_service.cc:1505] Processing CreateTablet for tablet ca0866077d3048208be1b85c29b29e11 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:37.295243 2709 tablet_service.cc:1505] Processing CreateTablet for tablet 8a775a3cea4f457aae26a552b402b7aa (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:37.295267 2786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca0866077d3048208be1b85c29b29e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.295336 2710 tablet_service.cc:1505] Processing CreateTablet for tablet ca0866077d3048208be1b85c29b29e11 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:37.295425 2711 tablet_service.cc:1505] Processing CreateTablet for tablet 483b1e0dd09347e885b38d9a012d22c5 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:37.295583 2635 tablet_service.cc:1505] Processing CreateTablet for tablet ca0866077d3048208be1b85c29b29e11 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251027 03:10:37.295646 2635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca0866077d3048208be1b85c29b29e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.296362 2633 tablet_service.cc:1505] Processing CreateTablet for tablet 8a775a3cea4f457aae26a552b402b7aa (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:37.296386 2787 tablet_service.cc:1505] Processing CreateTablet for tablet 483b1e0dd09347e885b38d9a012d22c5 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:37.296432 2633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a775a3cea4f457aae26a552b402b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.296448 2787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483b1e0dd09347e885b38d9a012d22c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.297397 2634 tablet_service.cc:1505] Processing CreateTablet for tablet 483b1e0dd09347e885b38d9a012d22c5 (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251027 03:10:37.297472 2634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483b1e0dd09347e885b38d9a012d22c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.298471 2849 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.298899 2849 tablet_bootstrap.cc:654] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.299238 2844 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.299499 2849 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.299542 2849 ts_tablet_manager.cc:1403] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.299609 2844 tablet_bootstrap.cc:654] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.299677 2849 raft_consensus.cc:359] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.299736 2849 raft_consensus.cc:385] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.299757 2849 raft_consensus.cc:740] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.299800 2849 consensus_queue.cc:260] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.295047 2785 tablet_service.cc:1505] Processing CreateTablet for tablet 8a775a3cea4f457aae26a552b402b7aa (DEFAULT_TABLE table=dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251027 03:10:37.299985 2785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a775a3cea4f457aae26a552b402b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.295198 2708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea846b9e34942c0a73ed0f05266a9f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.301327 2711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 483b1e0dd09347e885b38d9a012d22c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.301396 2709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a775a3cea4f457aae26a552b402b7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.302305 2845 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.302408 2710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca0866077d3048208be1b85c29b29e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:37.299911 2849 ts_tablet_manager.cc:1434] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.302592 2849 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.302712 2845 tablet_bootstrap.cc:654] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.302981 2849 tablet_bootstrap.cc:654] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.303290 2845 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.303346 2845 ts_tablet_manager.cc:1403] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.303503 2845 raft_consensus.cc:359] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.303562 2845 raft_consensus.cc:385] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.303586 2845 raft_consensus.cc:740] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.303617 2849 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.303627 2845 consensus_queue.cc:260] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.303684 2849 ts_tablet_manager.cc:1403] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:37.303776 2845 ts_tablet_manager.cc:1434] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:37.303849 2845 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.303887 2849 raft_consensus.cc:359] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.303951 2849 raft_consensus.cc:385] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.303979 2849 raft_consensus.cc:740] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.304036 2849 consensus_queue.cc:260] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.304124 2849 ts_tablet_manager.cc:1434] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.304205 2849 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.304245 2845 tablet_bootstrap.cc:654] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.304569 2849 tablet_bootstrap.cc:654] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.304888 2845 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.304937 2845 ts_tablet_manager.cc:1403] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.305104 2844 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.305158 2844 ts_tablet_manager.cc:1403] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20251027 03:10:37.305083 2845 raft_consensus.cc:359] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.305262 2845 raft_consensus.cc:385] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.305297 2844 raft_consensus.cc:359] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.305302 2845 raft_consensus.cc:740] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.305366 2844 raft_consensus.cc:385] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.305390 2845 consensus_queue.cc:260] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.305394 2844 raft_consensus.cc:740] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.305483 2844 consensus_queue.cc:260] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.305594 2844 ts_tablet_manager.cc:1434] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.305645 2849 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.305686 2849 ts_tablet_manager.cc:1403] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.305749 2844 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.305827 2849 raft_consensus.cc:359] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.305492 2845 ts_tablet_manager.cc:1434] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:37.305887 2849 raft_consensus.cc:385] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.305922 2849 raft_consensus.cc:740] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.305932 2845 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.305968 2849 consensus_queue.cc:260] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.306046 2849 ts_tablet_manager.cc:1434] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:37.306100 2849 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:37.306169 2844 tablet_bootstrap.cc:654] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.306342 2845 tablet_bootstrap.cc:654] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.306499 2849 tablet_bootstrap.cc:654] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.306788 2844 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.306834 2844 ts_tablet_manager.cc:1403] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.306986 2844 raft_consensus.cc:359] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.307044 2844 raft_consensus.cc:385] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.307066 2844 raft_consensus.cc:740] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.307109 2849 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: No bootstrap required, opened a new log
I20251027 03:10:37.307152 2849 ts_tablet_manager.cc:1403] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:37.307291 2849 raft_consensus.cc:359] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.307356 2849 raft_consensus.cc:385] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.307376 2849 raft_consensus.cc:740] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.307420 2849 consensus_queue.cc:260] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.307471 2845 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.307514 2845 ts_tablet_manager.cc:1403] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:10:37.307523 2849 ts_tablet_manager.cc:1434] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.307116 2844 consensus_queue.cc:260] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.307669 2844 ts_tablet_manager.cc:1434] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:37.307730 2844 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.307791 2845 raft_consensus.cc:359] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.307855 2845 raft_consensus.cc:385] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.307878 2845 raft_consensus.cc:740] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.308127 2844 tablet_bootstrap.cc:654] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.308141 2845 consensus_queue.cc:260] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.308560 2845 ts_tablet_manager.cc:1434] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 03:10:37.308624 2845 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Bootstrap starting.
I20251027 03:10:37.308763 2844 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.308812 2844 ts_tablet_manager.cc:1403] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.308938 2844 raft_consensus.cc:359] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.308997 2844 raft_consensus.cc:385] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.309016 2844 raft_consensus.cc:740] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.309020 2845 tablet_bootstrap.cc:654] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.309064 2844 consensus_queue.cc:260] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.309154 2844 ts_tablet_manager.cc:1434] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.309208 2844 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Bootstrap starting.
I20251027 03:10:37.309602 2844 tablet_bootstrap.cc:654] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Neither blocks nor log segments found. Creating new log.
I20251027 03:10:37.309624 2845 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: No bootstrap required, opened a new log
I20251027 03:10:37.309670 2845 ts_tablet_manager.cc:1403] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.309818 2845 raft_consensus.cc:359] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.309878 2845 raft_consensus.cc:385] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.309900 2845 raft_consensus.cc:740] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.309948 2845 consensus_queue.cc:260] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.310043 2845 ts_tablet_manager.cc:1434] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.310253 2844 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: No bootstrap required, opened a new log
I20251027 03:10:37.310307 2844 ts_tablet_manager.cc:1403] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:37.310456 2844 raft_consensus.cc:359] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.310513 2844 raft_consensus.cc:385] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 03:10:37.310530 2844 raft_consensus.cc:740] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Initialized, Role: FOLLOWER
I20251027 03:10:37.310588 2844 consensus_queue.cc:260] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.310715 2844 ts_tablet_manager.cc:1434] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:37.312288 2848 raft_consensus.cc:493] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.312332 2848 raft_consensus.cc:515] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.312427 2848 leader_election.cc:290] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.312553 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0866077d3048208be1b85c29b29e11" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f" is_pre_election: true
I20251027 03:10:37.312615 2797 raft_consensus.cc:2468] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.312605 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0866077d3048208be1b85c29b29e11" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.312685 2645 raft_consensus.cc:2468] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.312757 2686 leader_election.cc:304] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85419970b418485aa1e22a2bd96db12f, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.312837 2848 raft_consensus.cc:2804] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.312880 2848 raft_consensus.cc:493] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.312893 2848 raft_consensus.cc:3060] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.313324 2848 raft_consensus.cc:515] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.313421 2848 leader_election.cc:290] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.313546 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0866077d3048208be1b85c29b29e11" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f"
I20251027 03:10:37.313547 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ca0866077d3048208be1b85c29b29e11" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.313609 2797 raft_consensus.cc:3060] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.313642 2645 raft_consensus.cc:3060] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.314070 2797 raft_consensus.cc:2468] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.314143 2645 raft_consensus.cc:2468] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.314227 2686 leader_election.cc:304] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85419970b418485aa1e22a2bd96db12f, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.314306 2848 raft_consensus.cc:2804] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.314347 2848 raft_consensus.cc:697] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Becoming Leader. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Running, Role: LEADER
I20251027 03:10:37.314395 2848 consensus_queue.cc:237] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.314858 2539 catalog_manager.cc:5649] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:37.327131 2855 raft_consensus.cc:493] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.327184 2855 raft_consensus.cc:515] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.327294 2855 leader_election.cc:290] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:37.327452 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea846b9e34942c0a73ed0f05266a9f9" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.327464 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea846b9e34942c0a73ed0f05266a9f9" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" is_pre_election: true
I20251027 03:10:37.327543 2645 raft_consensus.cc:2468] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 0.
I20251027 03:10:37.327549 2721 raft_consensus.cc:2468] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 0.
I20251027 03:10:37.327679 2762 leader_election.cc:304] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.327750 2855 raft_consensus.cc:2804] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.327795 2855 raft_consensus.cc:493] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.327824 2855 raft_consensus.cc:3060] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.328332 2855 raft_consensus.cc:515] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.328434 2855 leader_election.cc:290] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:37.328567 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea846b9e34942c0a73ed0f05266a9f9" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.328603 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3ea846b9e34942c0a73ed0f05266a9f9" candidate_uuid: "85419970b418485aa1e22a2bd96db12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7"
I20251027 03:10:37.328622 2645 raft_consensus.cc:3060] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.328647 2721 raft_consensus.cc:3060] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.329195 2645 raft_consensus.cc:2468] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 1.
I20251027 03:10:37.329355 2762 leader_election.cc:304] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.329361 2721 raft_consensus.cc:2468] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85419970b418485aa1e22a2bd96db12f in term 1.
I20251027 03:10:37.329463 2855 raft_consensus.cc:2804] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.329532 2855 raft_consensus.cc:697] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Becoming Leader. State: Replica: 85419970b418485aa1e22a2bd96db12f, State: Running, Role: LEADER
I20251027 03:10:37.329638 2855 consensus_queue.cc:237] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.330114 2539 catalog_manager.cc:5649] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f reported cstate change: term changed from 0 to 1, leader changed from <none> to 85419970b418485aa1e22a2bd96db12f (127.27.151.67). New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:37.377720 2848 raft_consensus.cc:493] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.377812 2848 raft_consensus.cc:515] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.377960 2848 leader_election.cc:290] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.378127 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "483b1e0dd09347e885b38d9a012d22c5" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f" is_pre_election: true
I20251027 03:10:37.378127 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "483b1e0dd09347e885b38d9a012d22c5" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d" is_pre_election: true
I20251027 03:10:37.378259 2797 raft_consensus.cc:2468] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.378261 2645 raft_consensus.cc:2468] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 0.
I20251027 03:10:37.378420 2686 leader_election.cc:304] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85419970b418485aa1e22a2bd96db12f, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.378515 2848 raft_consensus.cc:2804] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.378542 2848 raft_consensus.cc:493] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.378566 2848 raft_consensus.cc:3060] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.379134 2848 raft_consensus.cc:515] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.379244 2848 leader_election.cc:290] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Requested vote from peers 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.379380 2645 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "483b1e0dd09347e885b38d9a012d22c5" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e07265804fe4b3a9c8214c7df139f1d"
I20251027 03:10:37.379413 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "483b1e0dd09347e885b38d9a012d22c5" candidate_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f"
I20251027 03:10:37.379457 2645 raft_consensus.cc:3060] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.379467 2797 raft_consensus.cc:3060] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.380070 2645 raft_consensus.cc:2468] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.380070 2797 raft_consensus.cc:2468] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7f03efa0cee4eecbba43f88bd2724a7 in term 1.
I20251027 03:10:37.380255 2687 leader_election.cc:304] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.380367 2848 raft_consensus.cc:2804] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.380425 2848 raft_consensus.cc:697] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Becoming Leader. State: Replica: b7f03efa0cee4eecbba43f88bd2724a7, State: Running, Role: LEADER
I20251027 03:10:37.380470 2848 consensus_queue.cc:237] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.381060 2539 catalog_manager.cc:5649] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66). New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:37.615231 2855 consensus_queue.cc:1048] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.616652 2894 consensus_queue.cc:1048] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.633987 2848 consensus_queue.cc:1048] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.635040 2848 consensus_queue.cc:1048] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.645567 2852 raft_consensus.cc:493] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 03:10:37.645623 2852 raft_consensus.cc:515] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.645750 2852 leader_election.cc:290] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.648170 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a775a3cea4f457aae26a552b402b7aa" candidate_uuid: "0e07265804fe4b3a9c8214c7df139f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" is_pre_election: true
I20251027 03:10:37.648247 2721 raft_consensus.cc:2468] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e07265804fe4b3a9c8214c7df139f1d in term 0.
I20251027 03:10:37.648293 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a775a3cea4f457aae26a552b402b7aa" candidate_uuid: "0e07265804fe4b3a9c8214c7df139f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f" is_pre_election: true
I20251027 03:10:37.648371 2797 raft_consensus.cc:2468] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e07265804fe4b3a9c8214c7df139f1d in term 0.
I20251027 03:10:37.648365 2611 leader_election.cc:304] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, b7f03efa0cee4eecbba43f88bd2724a7; no voters:
I20251027 03:10:37.648506 2852 raft_consensus.cc:2804] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 03:10:37.648547 2852 raft_consensus.cc:493] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 03:10:37.648569 2852 raft_consensus.cc:3060] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.649147 2852 raft_consensus.cc:515] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.649273 2852 leader_election.cc:290] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [CANDIDATE]: Term 1 election: Requested vote from peers b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409), 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:37.649392 2797 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a775a3cea4f457aae26a552b402b7aa" candidate_uuid: "0e07265804fe4b3a9c8214c7df139f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85419970b418485aa1e22a2bd96db12f"
I20251027 03:10:37.649441 2797 raft_consensus.cc:3060] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.649863 2797 raft_consensus.cc:2468] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e07265804fe4b3a9c8214c7df139f1d in term 1.
I20251027 03:10:37.649966 2721 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8a775a3cea4f457aae26a552b402b7aa" candidate_uuid: "0e07265804fe4b3a9c8214c7df139f1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f03efa0cee4eecbba43f88bd2724a7"
I20251027 03:10:37.650014 2721 raft_consensus.cc:3060] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 03:10:37.650020 2610 leader_election.cc:304] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e07265804fe4b3a9c8214c7df139f1d, 85419970b418485aa1e22a2bd96db12f; no voters:
I20251027 03:10:37.650131 2852 raft_consensus.cc:2804] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Leader election won for term 1
I20251027 03:10:37.650225 2852 raft_consensus.cc:697] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 LEADER]: Becoming Leader. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Running, Role: LEADER
I20251027 03:10:37.650276 2852 consensus_queue.cc:237] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:37.650609 2721 raft_consensus.cc:2468] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e07265804fe4b3a9c8214c7df139f1d in term 1.
I20251027 03:10:37.650775 2539 catalog_manager.cc:5649] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65). New cstate: current_term: 1 leader_uuid: "0e07265804fe4b3a9c8214c7df139f1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:37.723160 2848 consensus_queue.cc:1048] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.724447 2848 consensus_queue.cc:1048] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.734053 28253 tablet_server.cc:178] TabletServer@127.27.151.65:0 shutting down...
I20251027 03:10:37.736799 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:37.736922 28253 tablet_replica.cc:333] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.736985 28253 raft_consensus.cc:2243] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.737026 28253 raft_consensus.cc:2272] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.737195 28253 tablet_replica.cc:333] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.737236 28253 raft_consensus.cc:2243] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.737272 28253 raft_consensus.cc:2272] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.737376 28253 tablet_replica.cc:333] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.737411 28253 raft_consensus.cc:2243] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.737444 28253 raft_consensus.cc:2272] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.737610 28253 tablet_replica.cc:333] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.737648 28253 raft_consensus.cc:2243] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.737676 28253 raft_consensus.cc:2272] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.737847 28253 tablet_replica.cc:333] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.737886 28253 raft_consensus.cc:2243] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.737929 28253 raft_consensus.cc:2272] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.738109 28253 tablet_replica.cc:333] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.738152 28253 raft_consensus.cc:2243] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.738186 28253 raft_consensus.cc:2272] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.738349 28253 tablet_replica.cc:333] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.738389 28253 raft_consensus.cc:2243] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:37.738433 28253 pending_rounds.cc:70] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Trying to abort 1 pending ops.
I20251027 03:10:37.738456 28253 pending_rounds.cc:77] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245875815919616 op_type: NO_OP noop_request { }
I20251027 03:10:37.738495 28253 raft_consensus.cc:2889] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:37.738524 28253 raft_consensus.cc:2272] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:37.738767 28253 tablet_replica.cc:333] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: stopping tablet replica
I20251027 03:10:37.738809 28253 raft_consensus.cc:2243] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:37.738840 28253 raft_consensus.cc:2272] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 03:10:37.738988 2762 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20251027 03:10:37.739281 2762 consensus_peers.cc:597] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:37.746858 2894 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.754779 28253 tablet_server.cc:195] TabletServer@127.27.151.65:0 shutdown complete.
I20251027 03:10:37.864584 2848 consensus_queue.cc:1048] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 03:10:37.866302 2687 consensus_peers.cc:597] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:37.888662 2762 consensus_peers.cc:597] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:37.925725 2894 consensus_queue.cc:1048] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:37.954059 2864 consensus_queue.cc:1048] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 03:10:37.955546 2687 consensus_peers.cc:597] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:38.045213 2687 consensus_peers.cc:597] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:38.151774 2762 consensus_peers.cc:597] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251027 03:10:38.181322 2687 consensus_peers.cc:597] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:38.430373 2899 consensus_queue.cc:579] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Leader has been unable to successfully communicate with peer 0e07265804fe4b3a9c8214c7df139f1d for more than 1 seconds (1.050s)
I20251027 03:10:38.663295 2868 consensus_queue.cc:579] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Leader has been unable to successfully communicate with peer 0e07265804fe4b3a9c8214c7df139f1d for more than 1 seconds (1.334s)
I20251027 03:10:38.674398 2917 consensus_queue.cc:579] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Leader has been unable to successfully communicate with peer 0e07265804fe4b3a9c8214c7df139f1d for more than 1 seconds (1.057s)
I20251027 03:10:38.759131 2540 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38958:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20251027 03:10:38.759222 2540 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38958:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20251027 03:10:38.760107 2540 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Sending DeleteTablet for 3 replicas of tablet 3ea846b9e34942c0a73ed0f05266a9f9
I20251027 03:10:38.760252 2540 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Sending DeleteTablet for 3 replicas of tablet 8a775a3cea4f457aae26a552b402b7aa
I20251027 03:10:38.760309 2540 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Sending DeleteTablet for 3 replicas of tablet ca0866077d3048208be1b85c29b29e11
I20251027 03:10:38.760365 2540 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: Sending DeleteTablet for 3 replicas of tablet 483b1e0dd09347e885b38d9a012d22c5
I20251027 03:10:38.760423 2785 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ea846b9e34942c0a73ed0f05266a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:42094
I20251027 03:10:38.760558 2709 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ea846b9e34942c0a73ed0f05266a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:35666
I20251027 03:10:38.760634 2925 tablet_replica.cc:333] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:38.760567 2787 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a775a3cea4f457aae26a552b402b7aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:42094
I20251027 03:10:38.760540 2708 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0866077d3048208be1b85c29b29e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:35666
I20251027 03:10:38.760589 2711 tablet_service.cc:1552] Processing DeleteTablet for tablet 483b1e0dd09347e885b38d9a012d22c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:35666
I20251027 03:10:38.760695 2925 raft_consensus.cc:2243] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:38.760574 2710 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a775a3cea4f457aae26a552b402b7aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:35666
I20251027 03:10:38.760792 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:10:38.760820 2925 raft_consensus.cc:2272] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.760836 2787 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0866077d3048208be1b85c29b29e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:42094
I20251027 03:10:38.760983 2785 tablet_service.cc:1552] Processing DeleteTablet for tablet 483b1e0dd09347e885b38d9a012d22c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:42094
I20251027 03:10:38.761042 2925 ts_tablet_manager.cc:1916] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.761070 2926 tablet_replica.cc:333] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:38.761183 2926 raft_consensus.cc:2243] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.761266 2926 raft_consensus.cc:2272] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 03:10:38.761515 2528 catalog_manager.cc:4712] TS 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a775a3cea4f457aae26a552b402b7aa: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111)
I20251027 03:10:38.761538 2926 ts_tablet_manager.cc:1916] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 03:10:38.762164 2927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:38.762357 2931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:38.762578 28253 server_base.cc:1047] running on GCE node
I20251027 03:10:38.762880 2925 ts_tablet_manager.cc:1929] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.762945 2925 log.cc:1199] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/3ea846b9e34942c0a73ed0f05266a9f9
I20251027 03:10:38.763163 2925 ts_tablet_manager.cc:1950] T 3ea846b9e34942c0a73ed0f05266a9f9 P 85419970b418485aa1e22a2bd96db12f: Deleting consensus metadata
I20251027 03:10:38.763283 2926 ts_tablet_manager.cc:1929] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.763350 2926 log.cc:1199] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/3ea846b9e34942c0a73ed0f05266a9f9
I20251027 03:10:38.763453 2925 tablet_replica.cc:333] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:38.763511 2527 catalog_manager.cc:4985] TS 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727): tablet 3ea846b9e34942c0a73ed0f05266a9f9 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.763517 2925 raft_consensus.cc:2243] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.763597 2925 raft_consensus.cc:2272] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.763698 2864 consensus_queue.cc:579] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Leader has been unable to successfully communicate with peer 0e07265804fe4b3a9c8214c7df139f1d for more than 1 seconds (1.039s)
I20251027 03:10:38.763742 2925 ts_tablet_manager.cc:1916] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251027 03:10:38.763703 2928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:38.764204 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:38.764241 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:38.764259 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534638764259 us; error 0 us; skew 500 ppm
I20251027 03:10:38.764873 28253 webserver.cc:492] Webserver started at http://127.27.151.68:45249/ using document root <none> and password file <none>
I20251027 03:10:38.764966 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:38.764986 2925 ts_tablet_manager.cc:1929] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 03:10:38.765012 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 03:10:38.765027 2925 log.cc:1199] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/8a775a3cea4f457aae26a552b402b7aa
I20251027 03:10:38.765061 28253 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 03:10:38.765198 2925 ts_tablet_manager.cc:1950] T 8a775a3cea4f457aae26a552b402b7aa P 85419970b418485aa1e22a2bd96db12f: Deleting consensus metadata
I20251027 03:10:38.763553 2926 ts_tablet_manager.cc:1950] T 3ea846b9e34942c0a73ed0f05266a9f9 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting consensus metadata
I20251027 03:10:38.765341 28253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root/instance:
uuid: "fc491cacf9c649a4899b8505da575e48"
format_stamp: "Formatted at 2025-10-27 03:10:38 on dist-test-slave-4cc8"
I20251027 03:10:38.765482 2925 tablet_replica.cc:333] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:38.765542 2527 catalog_manager.cc:4985] TS 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727): tablet 8a775a3cea4f457aae26a552b402b7aa (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.765579 2926 tablet_replica.cc:333] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:38.765604 2528 catalog_manager.cc:4985] TS b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409): tablet 3ea846b9e34942c0a73ed0f05266a9f9 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.765636 2926 raft_consensus.cc:2243] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:38.765607 2925 raft_consensus.cc:2243] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.765703 2925 raft_consensus.cc:2272] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.765717 2926 raft_consensus.cc:2272] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.765908 2926 ts_tablet_manager.cc:1916] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.766377 28253 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:38.766389 2925 ts_tablet_manager.cc:1916] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.766952 2936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.767103 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.767158 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root
uuid: "fc491cacf9c649a4899b8505da575e48"
format_stamp: "Formatted at 2025-10-27 03:10:38 on dist-test-slave-4cc8"
I20251027 03:10:38.767210 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:38.767225 2926 ts_tablet_manager.cc:1929] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.767263 2926 log.cc:1199] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/ca0866077d3048208be1b85c29b29e11
I20251027 03:10:38.767468 2926 ts_tablet_manager.cc:1950] T ca0866077d3048208be1b85c29b29e11 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting consensus metadata
I20251027 03:10:38.767505 2925 ts_tablet_manager.cc:1929] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.767561 2925 log.cc:1199] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/ca0866077d3048208be1b85c29b29e11
I20251027 03:10:38.767781 2925 ts_tablet_manager.cc:1950] T ca0866077d3048208be1b85c29b29e11 P 85419970b418485aa1e22a2bd96db12f: Deleting consensus metadata
I20251027 03:10:38.767807 2528 catalog_manager.cc:4985] TS b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409): tablet ca0866077d3048208be1b85c29b29e11 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.767827 2926 tablet_replica.cc:333] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:38.767876 2926 raft_consensus.cc:2243] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:38.767930 2926 raft_consensus.cc:2272] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.767983 2925 tablet_replica.cc:333] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:38.768042 2925 raft_consensus.cc:2243] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.768077 2925 raft_consensus.cc:2272] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.768128 2527 catalog_manager.cc:4985] TS 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727): tablet ca0866077d3048208be1b85c29b29e11 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.768280 2925 ts_tablet_manager.cc:1916] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.768286 2926 ts_tablet_manager.cc:1916] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.769284 2925 ts_tablet_manager.cc:1929] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.769299 2926 ts_tablet_manager.cc:1929] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.769356 2925 log.cc:1199] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-2-root/wals/483b1e0dd09347e885b38d9a012d22c5
I20251027 03:10:38.769362 2926 log.cc:1199] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/483b1e0dd09347e885b38d9a012d22c5
I20251027 03:10:38.769557 2925 ts_tablet_manager.cc:1950] T 483b1e0dd09347e885b38d9a012d22c5 P 85419970b418485aa1e22a2bd96db12f: Deleting consensus metadata
I20251027 03:10:38.769562 2926 ts_tablet_manager.cc:1950] T 483b1e0dd09347e885b38d9a012d22c5 P b7f03efa0cee4eecbba43f88bd2724a7: Deleting consensus metadata
I20251027 03:10:38.769773 2926 tablet_replica.cc:333] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:38.769814 2926 raft_consensus.cc:2243] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.769845 2926 raft_consensus.cc:2272] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.769919 2527 catalog_manager.cc:4985] TS 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727): tablet 483b1e0dd09347e885b38d9a012d22c5 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.769935 2926 ts_tablet_manager.cc:1916] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.770004 2528 catalog_manager.cc:4985] TS b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409): tablet 483b1e0dd09347e885b38d9a012d22c5 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.770906 2926 ts_tablet_manager.cc:1929] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 03:10:38.770960 2926 log.cc:1199] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/8a775a3cea4f457aae26a552b402b7aa
I20251027 03:10:38.771126 2926 ts_tablet_manager.cc:1950] T 8a775a3cea4f457aae26a552b402b7aa P b7f03efa0cee4eecbba43f88bd2724a7: Deleting consensus metadata
I20251027 03:10:38.771369 2528 catalog_manager.cc:4985] TS b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409): tablet 8a775a3cea4f457aae26a552b402b7aa (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.790115 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:38.790284 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:38.790387 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:38.790527 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 03:10:38.790553 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.790580 28253 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 03:10:38.790606 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.793064 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.68:33461
I20251027 03:10:38.793480 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251027 03:10:38.793586 2999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.68:33461 every 8 connection(s)
W20251027 03:10:38.794523 3002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 03:10:38.794608 3003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:38.794772 28253 server_base.cc:1047] running on GCE node
W20251027 03:10:38.794869 3006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 03:10:38.794983 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251027 03:10:38.795017 28253 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251027 03:10:38.795029 28253 hybrid_clock.cc:648] HybridClock initialized: now 1761534638795030 us; error 0 us; skew 500 ppm
I20251027 03:10:38.795585 28253 webserver.cc:492] Webserver started at http://127.27.151.65:40263/ using document root <none> and password file <none>
I20251027 03:10:38.795657 28253 fs_manager.cc:362] Metadata directory not provided
I20251027 03:10:38.795688 28253 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251027 03:10:38.796247 28253 fs_manager.cc:714] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:38.797078 3011 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.797106 3000 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35495
I20251027 03:10:38.797151 3000 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:38.797196 28253 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:38.797236 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root
uuid: "0e07265804fe4b3a9c8214c7df139f1d"
format_stamp: "Formatted at 2025-10-27 03:10:35 on dist-test-slave-4cc8"
I20251027 03:10:38.797262 3000 heartbeater.cc:507] Master 127.27.151.126:35495 requested a full tablet report, sending...
I20251027 03:10:38.797283 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 03:10:38.797458 2540 ts_manager.cc:194] Registered new tserver with Master: fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461)
I20251027 03:10:38.797772 2540 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38992
I20251027 03:10:38.804358 2721 consensus_queue.cc:237] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:38.805002 2797 raft_consensus.cc:1275] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:38.805137 2899 consensus_queue.cc:1048] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251027 03:10:38.805438 2687 consensus_peers.cc:597] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): Couldn't send request to peer 0e07265804fe4b3a9c8214c7df139f1d. Status: Network error: Client connection negotiation failed: client connection to 127.27.151.65:37617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:38.805727 2848 raft_consensus.cc:2955] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.805824 2797 raft_consensus.cc:2955] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.806649 2528 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 201f9e8ee9304d1585da90d7f2f0d272 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
I20251027 03:10:38.806695 2540 catalog_manager.cc:5649] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f reported cstate change: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
W20251027 03:10:38.808316 2686 consensus_peers.cc:597] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): Couldn't send request to peer fc491cacf9c649a4899b8505da575e48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 201f9e8ee9304d1585da90d7f2f0d272. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:38.819260 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 03:10:38.819378 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 03:10:38.819487 28253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 03:10:38.819705 3022 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20251027 03:10:38.821635 28253 ts_tablet_manager.cc:585] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20251027 03:10:38.821681 28253 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20251027 03:10:38.821717 28253 ts_tablet_manager.cc:600] Registering tablets (0/8 complete)
I20251027 03:10:38.822108 3022 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.823040 3022 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.823189 3022 tablet_bootstrap.cc:492] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.823277 3022 ts_tablet_manager.cc:1403] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:38.823377 3022 raft_consensus.cc:359] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.823431 3022 raft_consensus.cc:740] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.823467 3022 consensus_queue.cc:260] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.823597 3022 ts_tablet_manager.cc:1434] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:38.823663 3022 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.823783 28253 ts_tablet_manager.cc:616] Registered 8 tablets
I20251027 03:10:38.823819 28253 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20251027 03:10:38.824453 3022 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.824594 3022 tablet_bootstrap.cc:492] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.824712 3022 ts_tablet_manager.cc:1403] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:38.824821 3022 raft_consensus.cc:359] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.824882 3022 raft_consensus.cc:740] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.824924 3022 consensus_queue.cc:260] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.825001 3022 ts_tablet_manager.cc:1434] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:38.825049 3022 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.825784 3022 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.826043 3022 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.826175 3022 ts_tablet_manager.cc:1403] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:38.826296 3022 raft_consensus.cc:359] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.826359 3022 raft_consensus.cc:740] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.826400 3022 consensus_queue.cc:260] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.826480 3022 ts_tablet_manager.cc:1434] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:38.826525 3022 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.826841 28253 rpc_server.cc:307] RPC server started. Bound to: 127.27.151.65:37617
I20251027 03:10:38.826911 3084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.151.65:37617 every 8 connection(s)
I20251027 03:10:38.827683 3022 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.827893 3022 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.827977 3022 ts_tablet_manager.cc:1403] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:38.828110 3022 raft_consensus.cc:359] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.828187 3022 raft_consensus.cc:740] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.828227 3022 consensus_queue.cc:260] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.828315 3022 ts_tablet_manager.cc:1434] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:38.828359 3022 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.829695 3022 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.829824 3085 heartbeater.cc:344] Connected to a master server at 127.27.151.126:35495
I20251027 03:10:38.829875 3085 heartbeater.cc:461] Registering TS with master...
I20251027 03:10:38.829900 3022 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.829957 3085 heartbeater.cc:507] Master 127.27.151.126:35495 requested a full tablet report, sending...
I20251027 03:10:38.829989 3022 ts_tablet_manager.cc:1403] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251027 03:10:38.830122 3022 raft_consensus.cc:359] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.830164 3022 raft_consensus.cc:740] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.830245 3022 consensus_queue.cc:260] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.830336 2539 ts_manager.cc:194] Re-registered known tserver with Master: 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617)
I20251027 03:10:38.830355 3022 ts_tablet_manager.cc:1434] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:38.830448 3022 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.830709 2539 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39006
I20251027 03:10:38.831029 3085 heartbeater.cc:499] Master 127.27.151.126:35495 was elected leader, sending a full tablet report...
I20251027 03:10:38.831570 3022 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.831801 3022 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.831889 3022 ts_tablet_manager.cc:1403] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:38.832023 3022 raft_consensus.cc:359] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.832073 3022 raft_consensus.cc:740] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.832120 3022 consensus_queue.cc:260] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.832214 3022 ts_tablet_manager.cc:1434] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20251027 03:10:38.832257 3022 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.833271 3022 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:38.833415 3022 tablet_bootstrap.cc:492] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.833500 3022 ts_tablet_manager.cc:1403] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:38.833643 3022 raft_consensus.cc:359] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.833712 3022 raft_consensus.cc:740] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.833750 3022 consensus_queue.cc:260] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.833849 3022 ts_tablet_manager.cc:1434] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.833897 3022 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap starting.
I20251027 03:10:38.834926 3022 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251027 03:10:38.835106 3022 tablet_bootstrap.cc:492] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Bootstrap complete.
I20251027 03:10:38.835189 3022 ts_tablet_manager.cc:1403] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:38.835328 3022 raft_consensus.cc:359] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.835374 3022 raft_consensus.cc:740] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e07265804fe4b3a9c8214c7df139f1d, State: Initialized, Role: FOLLOWER
I20251027 03:10:38.835417 3022 consensus_queue.cc:260] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:38.835516 3022 ts_tablet_manager.cc:1434] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:38.846865 3040 tablet_service.cc:1552] Processing DeleteTablet for tablet 483b1e0dd09347e885b38d9a012d22c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:45728
I20251027 03:10:38.847033 3094 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:38.847085 3094 raft_consensus.cc:2243] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.847115 3094 raft_consensus.cc:2272] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.847301 3094 ts_tablet_manager.cc:1916] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.848606 3094 ts_tablet_manager.cc:1929] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 03:10:38.848650 3094 log.cc:1199] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/483b1e0dd09347e885b38d9a012d22c5
I20251027 03:10:38.848806 3094 ts_tablet_manager.cc:1950] T 483b1e0dd09347e885b38d9a012d22c5 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting consensus metadata
I20251027 03:10:38.849095 2528 catalog_manager.cc:4985] TS 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): tablet 483b1e0dd09347e885b38d9a012d22c5 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.852492 2797 consensus_queue.cc:237] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:38.853188 2721 raft_consensus.cc:1275] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:38.853513 3040 tablet_service.cc:1552] Processing DeleteTablet for tablet 8a775a3cea4f457aae26a552b402b7aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:45728
I20251027 03:10:38.853596 3094 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:38.853559 2868 consensus_queue.cc:1048] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:10:38.853639 3094 raft_consensus.cc:2243] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.853685 3094 pending_rounds.cc:70] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Trying to abort 1 pending ops.
I20251027 03:10:38.853704 3094 pending_rounds.cc:77] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215245875815919616 op_type: NO_OP noop_request { }
I20251027 03:10:38.853742 3094 raft_consensus.cc:2889] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251027 03:10:38.853762 3094 raft_consensus.cc:2272] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.853964 3094 ts_tablet_manager.cc:1916] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.854596 3040 tablet_service.cc:1552] Processing DeleteTablet for tablet ca0866077d3048208be1b85c29b29e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:45728
W20251027 03:10:38.855556 2761 consensus_peers.cc:597] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f -> Peer fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): Couldn't send request to peer fc491cacf9c649a4899b8505da575e48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd9a239c4fa148f981838fd3d0884c50. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:38.855507 2916 raft_consensus.cc:2955] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.856107 2721 raft_consensus.cc:2955] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.856166 3058 raft_consensus.cc:1275] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:38.856375 2917 consensus_queue.cc:1048] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:10:38.856670 2527 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd9a239c4fa148f981838fd3d0884c50 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20251027 03:10:38.856734 2539 catalog_manager.cc:5649] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f reported cstate change: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:38.856976 3058 raft_consensus.cc:2955] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.857240 3040 tablet_service.cc:1552] Processing DeleteTablet for tablet 3ea846b9e34942c0a73ed0f05266a9f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 03:10:38 UTC) from {username='slave'} at 127.0.0.1:45728
I20251027 03:10:38.857301 3094 ts_tablet_manager.cc:1929] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251027 03:10:38.857345 3094 log.cc:1199] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/8a775a3cea4f457aae26a552b402b7aa
I20251027 03:10:38.857570 3094 ts_tablet_manager.cc:1950] T 8a775a3cea4f457aae26a552b402b7aa P 0e07265804fe4b3a9c8214c7df139f1d: Deleting consensus metadata
I20251027 03:10:38.857776 3094 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:38.857824 3094 raft_consensus.cc:2243] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.857824 2528 catalog_manager.cc:4985] TS 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): tablet 8a775a3cea4f457aae26a552b402b7aa (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.857867 3094 raft_consensus.cc:2272] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.858160 3094 ts_tablet_manager.cc:1916] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.859110 3094 ts_tablet_manager.cc:1929] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 03:10:38.859170 3094 log.cc:1199] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/ca0866077d3048208be1b85c29b29e11
I20251027 03:10:38.859297 3094 ts_tablet_manager.cc:1950] T ca0866077d3048208be1b85c29b29e11 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting consensus metadata
I20251027 03:10:38.859495 3094 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:38.859537 3094 raft_consensus.cc:2243] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:38.859568 2528 catalog_manager.cc:4985] TS 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): tablet ca0866077d3048208be1b85c29b29e11 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.859567 3094 raft_consensus.cc:2272] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:38.859769 3094 ts_tablet_manager.cc:1916] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 03:10:38.860493 3094 ts_tablet_manager.cc:1929] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251027 03:10:38.860538 3094 log.cc:1199] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-0-root/wals/3ea846b9e34942c0a73ed0f05266a9f9
I20251027 03:10:38.860703 3094 ts_tablet_manager.cc:1950] T 3ea846b9e34942c0a73ed0f05266a9f9 P 0e07265804fe4b3a9c8214c7df139f1d: Deleting consensus metadata
I20251027 03:10:38.860972 2528 catalog_manager.cc:4985] TS 0e07265804fe4b3a9c8214c7df139f1d (127.27.151.65:37617): tablet 3ea846b9e34942c0a73ed0f05266a9f9 (table dugtrio [id=4d209e9e7ca5409fa38e8551be3c630d]) successfully deleted
I20251027 03:10:38.952334 2797 consensus_queue.cc:237] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:38.952853 3058 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:38.952862 2721 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251027 03:10:38.952947 2761 consensus_peers.cc:597] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f -> Peer fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): Couldn't send request to peer fc491cacf9c649a4899b8505da575e48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5293919b0184807b8b08ab4ba8bd2ec. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:38.953131 2917 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 03:10:38.953212 2916 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:10:38.953868 2917 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.953985 2721 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.954262 3058 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.955260 2540 catalog_manager.cc:5649] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f reported cstate change: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251027 03:10:38.958082 2590 auto_rebalancer.cc:253] failed to send replica move request: Invalid argument: Server with UUID fc491cacf9c649a4899b8505da575e48 is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
W20251027 03:10:38.958622 2590 auto_rebalancer.cc:663] Could not move replica: Illegal state: 201f9e8ee9304d1585da90d7f2f0d272: source replica 85419970b418485aa1e22a2bd96db12f does not have REPLACE attribute set
W20251027 03:10:38.958659 2590 auto_rebalancer.cc:263] scheduled replica move failed to complete: Illegal state: 201f9e8ee9304d1585da90d7f2f0d272: source replica 85419970b418485aa1e22a2bd96db12f does not have REPLACE attribute set
W20251027 03:10:38.959123 2590 auto_rebalancer.cc:663] Could not move replica: Illegal state: bd9a239c4fa148f981838fd3d0884c50: source replica 0e07265804fe4b3a9c8214c7df139f1d does not have REPLACE attribute set
W20251027 03:10:38.959149 2590 auto_rebalancer.cc:263] scheduled replica move failed to complete: Illegal state: bd9a239c4fa148f981838fd3d0884c50: source replica 0e07265804fe4b3a9c8214c7df139f1d does not have REPLACE attribute set
I20251027 03:10:38.995793 2899 consensus_queue.cc:579] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Leader has been unable to successfully communicate with peer 0e07265804fe4b3a9c8214c7df139f1d for more than 1 seconds (1.360s)
I20251027 03:10:38.996516 2721 consensus_queue.cc:237] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
W20251027 03:10:38.997123 2686 consensus_peers.cc:597] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 -> Peer fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): Couldn't send request to peer fc491cacf9c649a4899b8505da575e48. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d7622b5f498448088e0344b369cdaa4. This is attempt 1: this message will repeat every 5th retry.
I20251027 03:10:38.997195 2797 raft_consensus.cc:1275] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:38.997372 2899 consensus_queue.cc:1048] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:10:38.998193 2864 raft_consensus.cc:2955] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.999331 3057 raft_consensus.cc:1275] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 03:10:38.999368 2528 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6d7622b5f498448088e0344b369cdaa4 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251027 03:10:38.999550 2848 consensus_queue.cc:1048] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251027 03:10:38.999636 2797 raft_consensus.cc:2955] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:38.999688 2539 catalog_manager.cc:5649] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251027 03:10:39.000062 3057 raft_consensus.cc:2955] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:39.238319 3110 ts_tablet_manager.cc:933] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Initiating tablet copy from peer 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:39.238554 3110 tablet_copy_client.cc:323] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:45727
I20251027 03:10:39.241309 2807 tablet_copy_service.cc:140] P 85419970b418485aa1e22a2bd96db12f: Received BeginTabletCopySession request for tablet bd9a239c4fa148f981838fd3d0884c50 from peer fc491cacf9c649a4899b8505da575e48 ({username='slave'} at 127.0.0.1:42132)
I20251027 03:10:39.241369 2807 tablet_copy_service.cc:161] P 85419970b418485aa1e22a2bd96db12f: Beginning new tablet copy session on tablet bd9a239c4fa148f981838fd3d0884c50 from peer fc491cacf9c649a4899b8505da575e48 at {username='slave'} at 127.0.0.1:42132: session id = fc491cacf9c649a4899b8505da575e48-bd9a239c4fa148f981838fd3d0884c50
I20251027 03:10:39.241814 2807 tablet_copy_source_session.cc:215] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:39.242127 3110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9a239c4fa148f981838fd3d0884c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:39.243348 3110 tablet_copy_client.cc:806] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:39.243463 3110 tablet_copy_client.cc:670] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:39.244059 3110 tablet_copy_client.cc:538] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:39.244585 3110 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Bootstrap starting.
I20251027 03:10:39.245771 3110 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:39.245924 3110 tablet_bootstrap.cc:492] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Bootstrap complete.
I20251027 03:10:39.246011 3110 ts_tablet_manager.cc:1403] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20251027 03:10:39.246160 3110 raft_consensus.cc:359] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.246217 3110 raft_consensus.cc:740] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc491cacf9c649a4899b8505da575e48, State: Initialized, Role: LEARNER
I20251027 03:10:39.246248 3110 consensus_queue.cc:260] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.246397 3110 ts_tablet_manager.cc:1434] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20251027 03:10:39.246433 3000 heartbeater.cc:499] Master 127.27.151.126:35495 was elected leader, sending a full tablet report...
I20251027 03:10:39.246521 2807 tablet_copy_service.cc:342] P 85419970b418485aa1e22a2bd96db12f: Request end of tablet copy session fc491cacf9c649a4899b8505da575e48-bd9a239c4fa148f981838fd3d0884c50 received from {username='slave'} at 127.0.0.1:42132
I20251027 03:10:39.246562 2807 tablet_copy_service.cc:434] P 85419970b418485aa1e22a2bd96db12f: ending tablet copy session fc491cacf9c649a4899b8505da575e48-bd9a239c4fa148f981838fd3d0884c50 on tablet bd9a239c4fa148f981838fd3d0884c50 with peer fc491cacf9c649a4899b8505da575e48
I20251027 03:10:39.332559 3057 raft_consensus.cc:2955] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } } }
I20251027 03:10:39.382854 3110 ts_tablet_manager.cc:933] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Initiating tablet copy from peer b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:39.383052 3110 tablet_copy_client.cc:323] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:46409
I20251027 03:10:39.385612 2796 consensus_queue.cc:237] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:39.386171 2721 raft_consensus.cc:1275] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:39.386227 3057 raft_consensus.cc:1275] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:39.386399 2916 consensus_queue.cc:1048] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:39.386397 2731 tablet_copy_service.cc:140] P b7f03efa0cee4eecbba43f88bd2724a7: Received BeginTabletCopySession request for tablet 201f9e8ee9304d1585da90d7f2f0d272 from peer fc491cacf9c649a4899b8505da575e48 ({username='slave'} at 127.0.0.1:35710)
I20251027 03:10:39.386492 2731 tablet_copy_service.cc:161] P b7f03efa0cee4eecbba43f88bd2724a7: Beginning new tablet copy session on tablet 201f9e8ee9304d1585da90d7f2f0d272 from peer fc491cacf9c649a4899b8505da575e48 at {username='slave'} at 127.0.0.1:35710: session id = fc491cacf9c649a4899b8505da575e48-201f9e8ee9304d1585da90d7f2f0d272
I20251027 03:10:39.386530 2868 consensus_queue.cc:1048] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:39.387043 2916 raft_consensus.cc:2955] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.387161 2731 tablet_copy_source_session.cc:215] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:39.387941 2527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet bd9a239c4fa148f981838fd3d0884c50 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:39.387931 3057 raft_consensus.cc:2955] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.388101 3110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201f9e8ee9304d1585da90d7f2f0d272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:39.388167 2539 catalog_manager.cc:5649] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f reported cstate change: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:39.388373 2721 raft_consensus.cc:2955] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.389369 3110 tablet_copy_client.cc:806] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:39.389470 3110 tablet_copy_client.cc:670] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:39.390194 3110 tablet_copy_client.cc:538] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:39.390867 3110 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Bootstrap starting.
I20251027 03:10:39.392048 3110 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:39.392253 3110 tablet_bootstrap.cc:492] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Bootstrap complete.
I20251027 03:10:39.392349 3110 ts_tablet_manager.cc:1403] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20251027 03:10:39.392475 3110 raft_consensus.cc:359] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.392546 3110 raft_consensus.cc:740] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc491cacf9c649a4899b8505da575e48, State: Initialized, Role: LEARNER
I20251027 03:10:39.392583 3110 consensus_queue.cc:260] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.392694 3110 ts_tablet_manager.cc:1434] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:39.392926 2731 tablet_copy_service.cc:342] P b7f03efa0cee4eecbba43f88bd2724a7: Request end of tablet copy session fc491cacf9c649a4899b8505da575e48-201f9e8ee9304d1585da90d7f2f0d272 received from {username='slave'} at 127.0.0.1:35710
I20251027 03:10:39.392973 2731 tablet_copy_service.cc:434] P b7f03efa0cee4eecbba43f88bd2724a7: ending tablet copy session fc491cacf9c649a4899b8505da575e48-201f9e8ee9304d1585da90d7f2f0d272 on tablet 201f9e8ee9304d1585da90d7f2f0d272 with peer fc491cacf9c649a4899b8505da575e48
I20251027 03:10:39.393386 2964 tablet_service.cc:1552] Processing DeleteTablet for tablet bd9a239c4fa148f981838fd3d0884c50 with delete_type TABLET_DATA_TOMBSTONED (TS fc491cacf9c649a4899b8505da575e48 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58312
I20251027 03:10:39.393528 3119 tablet_replica.cc:333] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: stopping tablet replica
I20251027 03:10:39.393582 3119 raft_consensus.cc:2243] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Raft consensus shutting down.
I20251027 03:10:39.393635 3119 raft_consensus.cc:2272] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Raft consensus is shut down!
I20251027 03:10:39.393826 3119 ts_tablet_manager.cc:1916] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:10:39.395131 3119 ts_tablet_manager.cc:1929] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20251027 03:10:39.395187 3119 log.cc:1199] T bd9a239c4fa148f981838fd3d0884c50 P fc491cacf9c649a4899b8505da575e48: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root/wals/bd9a239c4fa148f981838fd3d0884c50
I20251027 03:10:39.395496 2527 catalog_manager.cc:4985] TS fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): tablet bd9a239c4fa148f981838fd3d0884c50 (table test-workload [id=164795683cff4e139e24f3460173cb15]) successfully deleted
I20251027 03:10:39.523445 3110 ts_tablet_manager.cc:933] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: Initiating tablet copy from peer 85419970b418485aa1e22a2bd96db12f (127.27.151.67:45727)
I20251027 03:10:39.523648 3110 tablet_copy_client.cc:323] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.67:45727
I20251027 03:10:39.523844 2807 tablet_copy_service.cc:140] P 85419970b418485aa1e22a2bd96db12f: Received BeginTabletCopySession request for tablet f5293919b0184807b8b08ab4ba8bd2ec from peer fc491cacf9c649a4899b8505da575e48 ({username='slave'} at 127.0.0.1:42132)
I20251027 03:10:39.523895 2807 tablet_copy_service.cc:161] P 85419970b418485aa1e22a2bd96db12f: Beginning new tablet copy session on tablet f5293919b0184807b8b08ab4ba8bd2ec from peer fc491cacf9c649a4899b8505da575e48 at {username='slave'} at 127.0.0.1:42132: session id = fc491cacf9c649a4899b8505da575e48-f5293919b0184807b8b08ab4ba8bd2ec
I20251027 03:10:39.524468 2807 tablet_copy_source_session.cc:215] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:39.524756 3110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5293919b0184807b8b08ab4ba8bd2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:39.525910 3110 tablet_copy_client.cc:806] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:39.526028 3110 tablet_copy_client.cc:670] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:39.526540 3110 tablet_copy_client.cc:538] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:39.527227 3110 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: Bootstrap starting.
I20251027 03:10:39.528625 3110 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:39.528831 3110 tablet_bootstrap.cc:492] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: Bootstrap complete.
I20251027 03:10:39.528952 3110 ts_tablet_manager.cc:1403] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20251027 03:10:39.529130 3110 raft_consensus.cc:359] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.529213 3110 raft_consensus.cc:740] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc491cacf9c649a4899b8505da575e48, State: Initialized, Role: LEARNER
I20251027 03:10:39.529307 3110 consensus_queue.cc:260] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.529446 3110 ts_tablet_manager.cc:1434] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:39.529579 2807 tablet_copy_service.cc:342] P 85419970b418485aa1e22a2bd96db12f: Request end of tablet copy session fc491cacf9c649a4899b8505da575e48-f5293919b0184807b8b08ab4ba8bd2ec received from {username='slave'} at 127.0.0.1:42132
I20251027 03:10:39.529639 2807 tablet_copy_service.cc:434] P 85419970b418485aa1e22a2bd96db12f: ending tablet copy session fc491cacf9c649a4899b8505da575e48-f5293919b0184807b8b08ab4ba8bd2ec on tablet f5293919b0184807b8b08ab4ba8bd2ec with peer fc491cacf9c649a4899b8505da575e48
I20251027 03:10:39.531407 3110 ts_tablet_manager.cc:933] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Initiating tablet copy from peer b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409)
I20251027 03:10:39.531541 3110 tablet_copy_client.cc:323] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: tablet copy: Beginning tablet copy session from remote peer at address 127.27.151.66:46409
I20251027 03:10:39.531716 2731 tablet_copy_service.cc:140] P b7f03efa0cee4eecbba43f88bd2724a7: Received BeginTabletCopySession request for tablet 6d7622b5f498448088e0344b369cdaa4 from peer fc491cacf9c649a4899b8505da575e48 ({username='slave'} at 127.0.0.1:35710)
I20251027 03:10:39.531776 2731 tablet_copy_service.cc:161] P b7f03efa0cee4eecbba43f88bd2724a7: Beginning new tablet copy session on tablet 6d7622b5f498448088e0344b369cdaa4 from peer fc491cacf9c649a4899b8505da575e48 at {username='slave'} at 127.0.0.1:35710: session id = fc491cacf9c649a4899b8505da575e48-6d7622b5f498448088e0344b369cdaa4
I20251027 03:10:39.532198 2731 tablet_copy_source_session.cc:215] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 03:10:39.532536 3110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d7622b5f498448088e0344b369cdaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 03:10:39.533558 3110 tablet_copy_client.cc:806] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 0 data blocks...
I20251027 03:10:39.533712 3110 tablet_copy_client.cc:670] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: tablet copy: Starting download of 1 WAL segments...
I20251027 03:10:39.534250 3110 tablet_copy_client.cc:538] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 03:10:39.534910 3110 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Bootstrap starting.
I20251027 03:10:39.536291 3110 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 03:10:39.536485 3110 tablet_bootstrap.cc:492] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Bootstrap complete.
I20251027 03:10:39.536579 3110 ts_tablet_manager.cc:1403] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20251027 03:10:39.536746 3110 raft_consensus.cc:359] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.536809 3110 raft_consensus.cc:740] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fc491cacf9c649a4899b8505da575e48, State: Initialized, Role: LEARNER
I20251027 03:10:39.536829 3110 consensus_queue.cc:260] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: NON_VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: true } }
I20251027 03:10:39.536903 3110 ts_tablet_manager.cc:1434] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20251027 03:10:39.537051 2731 tablet_copy_service.cc:342] P b7f03efa0cee4eecbba43f88bd2724a7: Request end of tablet copy session fc491cacf9c649a4899b8505da575e48-6d7622b5f498448088e0344b369cdaa4 received from {username='slave'} at 127.0.0.1:35710
I20251027 03:10:39.537102 2731 tablet_copy_service.cc:434] P b7f03efa0cee4eecbba43f88bd2724a7: ending tablet copy session fc491cacf9c649a4899b8505da575e48-6d7622b5f498448088e0344b369cdaa4 on tablet 6d7622b5f498448088e0344b369cdaa4 with peer fc491cacf9c649a4899b8505da575e48
I20251027 03:10:39.546983 2721 consensus_queue.cc:237] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:39.547330 2796 raft_consensus.cc:1275] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:39.547408 3057 raft_consensus.cc:1275] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:39.547523 2848 consensus_queue.cc:1048] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:39.547612 3124 consensus_queue.cc:1048] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:39.548245 2848 raft_consensus.cc:2955] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.548359 3057 raft_consensus.cc:2955] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.548370 2797 raft_consensus.cc:2955] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.549338 2540 catalog_manager.cc:5649] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } health_report { overall_health: HEALTHY } } }
I20251027 03:10:39.549579 2528 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6d7622b5f498448088e0344b369cdaa4 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:39.550045 2964 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d7622b5f498448088e0344b369cdaa4 with delete_type TABLET_DATA_TOMBSTONED (TS fc491cacf9c649a4899b8505da575e48 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58312
I20251027 03:10:39.550143 3119 tablet_replica.cc:333] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: stopping tablet replica
I20251027 03:10:39.550207 3119 raft_consensus.cc:2243] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Raft consensus shutting down.
I20251027 03:10:39.550240 3119 raft_consensus.cc:2272] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Raft consensus is shut down!
I20251027 03:10:39.550406 3119 ts_tablet_manager.cc:1916] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:10:39.551501 3119 ts_tablet_manager.cc:1929] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20251027 03:10:39.551555 3119 log.cc:1199] T 6d7622b5f498448088e0344b369cdaa4 P fc491cacf9c649a4899b8505da575e48: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root/wals/6d7622b5f498448088e0344b369cdaa4
I20251027 03:10:39.551833 2527 catalog_manager.cc:4985] TS fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): tablet 6d7622b5f498448088e0344b369cdaa4 (table test-workload [id=164795683cff4e139e24f3460173cb15]) successfully deleted
I20251027 03:10:39.825616 2721 consensus_queue.cc:237] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } }
I20251027 03:10:39.826117 3057 raft_consensus.cc:1275] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:39.826138 2797 raft_consensus.cc:1275] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Refusing update from remote peer b7f03efa0cee4eecbba43f88bd2724a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:39.826296 2848 consensus_queue.cc:1048] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:39.826370 3124 consensus_queue.cc:1048] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:39.826967 3125 raft_consensus.cc:2955] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.827075 3057 raft_consensus.cc:2955] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.827811 2528 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 201f9e8ee9304d1585da90d7f2f0d272 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:39.827917 2797 raft_consensus.cc:2955] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.828079 2539 catalog_manager.cc:5649] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d reported cstate change: config changed from index 2 to 3, NON_VOTER fc491cacf9c649a4899b8505da575e48 (127.27.151.68) evicted. New cstate: current_term: 1 leader_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } }
I20251027 03:10:39.829016 2964 tablet_service.cc:1552] Processing DeleteTablet for tablet 201f9e8ee9304d1585da90d7f2f0d272 with delete_type TABLET_DATA_TOMBSTONED (TS fc491cacf9c649a4899b8505da575e48 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58312
I20251027 03:10:39.829116 3119 tablet_replica.cc:333] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: stopping tablet replica
I20251027 03:10:39.829182 3119 raft_consensus.cc:2243] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Raft consensus shutting down.
I20251027 03:10:39.829223 3119 raft_consensus.cc:2272] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Raft consensus is shut down!
I20251027 03:10:39.829421 3119 ts_tablet_manager.cc:1916] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:10:39.830838 3119 ts_tablet_manager.cc:1929] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20251027 03:10:39.830897 3119 log.cc:1199] T 201f9e8ee9304d1585da90d7f2f0d272 P fc491cacf9c649a4899b8505da575e48: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-3-root/wals/201f9e8ee9304d1585da90d7f2f0d272
I20251027 03:10:39.831236 2527 catalog_manager.cc:4985] TS fc491cacf9c649a4899b8505da575e48 (127.27.151.68:33461): tablet 201f9e8ee9304d1585da90d7f2f0d272 (table test-workload [id=164795683cff4e139e24f3460173cb15]) successfully deleted
I20251027 03:10:39.990165 2974 raft_consensus.cc:1217] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251027 03:10:40.439690 3128 raft_consensus.cc:1064] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: attempting to promote NON_VOTER fc491cacf9c649a4899b8505da575e48 to VOTER
I20251027 03:10:40.439935 3128 consensus_queue.cc:237] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } }
I20251027 03:10:40.440402 2974 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 LEARNER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:40.440441 2721 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:40.440558 3057 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251027 03:10:40.440790 3131 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:40.440886 3130 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:40.441038 3131 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251027 03:10:40.441781 3130 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fc491cacf9c649a4899b8505da575e48 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.441887 2721 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fc491cacf9c649a4899b8505da575e48 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.442739 2974 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fc491cacf9c649a4899b8505da575e48 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.442838 2541 catalog_manager.cc:5649] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 reported cstate change: config changed from index 2 to 3, fc491cacf9c649a4899b8505da575e48 (127.27.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.442852 3057 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, fc491cacf9c649a4899b8505da575e48 (127.27.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "b7f03efa0cee4eecbba43f88bd2724a7" member_type: VOTER last_known_addr { host: "127.27.151.66" port: 46409 } attrs { replace: true } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.444120 2797 consensus_queue.cc:237] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } }
I20251027 03:10:40.444509 2974 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:40.444680 3057 raft_consensus.cc:1275] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Refusing update from remote peer 85419970b418485aa1e22a2bd96db12f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251027 03:10:40.444680 3131 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:40.444842 2868 consensus_queue.cc:1048] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251027 03:10:40.445241 3131 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.445358 2974 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.445986 2527 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet f5293919b0184807b8b08ab4ba8bd2ec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251027 03:10:40.446070 3057 raft_consensus.cc:2955] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.446103 2539 catalog_manager.cc:5649] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 reported cstate change: config changed from index 3 to 4, VOTER b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66) evicted. New cstate: current_term: 1 leader_uuid: "85419970b418485aa1e22a2bd96db12f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0e07265804fe4b3a9c8214c7df139f1d" member_type: VOTER last_known_addr { host: "127.27.151.65" port: 37617 } } peers { permanent_uuid: "85419970b418485aa1e22a2bd96db12f" member_type: VOTER last_known_addr { host: "127.27.151.67" port: 45727 } } peers { permanent_uuid: "fc491cacf9c649a4899b8505da575e48" member_type: VOTER last_known_addr { host: "127.27.151.68" port: 33461 } attrs { promote: false } } }
I20251027 03:10:40.446723 2710 tablet_service.cc:1552] Processing DeleteTablet for tablet f5293919b0184807b8b08ab4ba8bd2ec with delete_type TABLET_DATA_TOMBSTONED (TS b7f03efa0cee4eecbba43f88bd2724a7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35666
I20251027 03:10:40.446976 3140 tablet_replica.cc:333] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:40.447036 3140 raft_consensus.cc:2243] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:40.447088 3140 raft_consensus.cc:2272] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:40.447355 3140 ts_tablet_manager.cc:1916] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251027 03:10:40.448393 3140 ts_tablet_manager.cc:1929] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251027 03:10:40.448460 3140 log.cc:1199] T f5293919b0184807b8b08ab4ba8bd2ec P b7f03efa0cee4eecbba43f88bd2724a7: Deleting WAL directory at /tmp/dist-test-taskAAGBjh/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761534461946371-28253-0/minicluster-data/ts-1-root/wals/f5293919b0184807b8b08ab4ba8bd2ec
I20251027 03:10:40.448740 2528 catalog_manager.cc:4985] TS b7f03efa0cee4eecbba43f88bd2724a7 (127.27.151.66:46409): tablet f5293919b0184807b8b08ab4ba8bd2ec (table test-workload [id=164795683cff4e139e24f3460173cb15]) successfully deleted
W20251027 03:10:41.446326 2590 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver fc491cacf9c649a4899b8505da575e48 not available for placement
I20251027 03:10:42.403650 28253 tablet_server.cc:178] TabletServer@127.27.151.65:37617 shutting down...
I20251027 03:10:42.406375 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:42.406514 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:42.406599 28253 raft_consensus.cc:2243] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.406648 28253 raft_consensus.cc:2272] T bd9a239c4fa148f981838fd3d0884c50 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.406813 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:42.406862 28253 raft_consensus.cc:2243] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.406917 28253 raft_consensus.cc:2272] T f5293919b0184807b8b08ab4ba8bd2ec P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.407025 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:42.407052 28253 raft_consensus.cc:2243] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.407078 28253 raft_consensus.cc:2272] T 6d7622b5f498448088e0344b369cdaa4 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.407147 28253 tablet_replica.cc:333] stopping tablet replica
I20251027 03:10:42.407171 28253 raft_consensus.cc:2243] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.407198 28253 raft_consensus.cc:2272] T 201f9e8ee9304d1585da90d7f2f0d272 P 0e07265804fe4b3a9c8214c7df139f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.418851 28253 tablet_server.cc:195] TabletServer@127.27.151.65:37617 shutdown complete.
I20251027 03:10:42.420254 28253 tablet_server.cc:178] TabletServer@127.27.151.66:0 shutting down...
I20251027 03:10:42.423676 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:42.423782 28253 tablet_replica.cc:333] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:42.423849 28253 raft_consensus.cc:2243] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:42.423915 28253 raft_consensus.cc:2272] T 6d7622b5f498448088e0344b369cdaa4 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.424057 28253 tablet_replica.cc:333] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:42.424098 28253 raft_consensus.cc:2243] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:42.424170 28253 raft_consensus.cc:2272] T 201f9e8ee9304d1585da90d7f2f0d272 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.424288 28253 tablet_replica.cc:333] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7: stopping tablet replica
I20251027 03:10:42.424320 28253 raft_consensus.cc:2243] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.424363 28253 raft_consensus.cc:2272] T bd9a239c4fa148f981838fd3d0884c50 P b7f03efa0cee4eecbba43f88bd2724a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.435988 28253 tablet_server.cc:195] TabletServer@127.27.151.66:0 shutdown complete.
I20251027 03:10:42.437147 28253 tablet_server.cc:178] TabletServer@127.27.151.67:0 shutting down...
I20251027 03:10:42.440779 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:42.440876 28253 tablet_replica.cc:333] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:42.440946 28253 raft_consensus.cc:2243] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.440999 28253 raft_consensus.cc:2272] T 6d7622b5f498448088e0344b369cdaa4 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.441135 28253 tablet_replica.cc:333] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:42.441170 28253 raft_consensus.cc:2243] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:42.441229 28253 raft_consensus.cc:2272] T f5293919b0184807b8b08ab4ba8bd2ec P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.441373 28253 tablet_replica.cc:333] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:42.441406 28253 raft_consensus.cc:2243] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.441452 28253 raft_consensus.cc:2272] T 201f9e8ee9304d1585da90d7f2f0d272 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.441555 28253 tablet_replica.cc:333] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f: stopping tablet replica
I20251027 03:10:42.441588 28253 raft_consensus.cc:2243] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:42.441658 28253 raft_consensus.cc:2272] T bd9a239c4fa148f981838fd3d0884c50 P 85419970b418485aa1e22a2bd96db12f [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 03:10:42.446533 2590 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver b7f03efa0cee4eecbba43f88bd2724a7 not available for placement
I20251027 03:10:42.453935 28253 tablet_server.cc:195] TabletServer@127.27.151.67:0 shutdown complete.
I20251027 03:10:42.454958 28253 tablet_server.cc:178] TabletServer@127.27.151.68:0 shutting down...
I20251027 03:10:42.457728 28253 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251027 03:10:42.457834 28253 tablet_replica.cc:333] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48: stopping tablet replica
I20251027 03:10:42.457885 28253 raft_consensus.cc:2243] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 03:10:42.457929 28253 raft_consensus.cc:2272] T f5293919b0184807b8b08ab4ba8bd2ec P fc491cacf9c649a4899b8505da575e48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.469288 28253 tablet_server.cc:195] TabletServer@127.27.151.68:0 shutdown complete.
I20251027 03:10:42.470196 28253 master.cc:561] Master@127.27.151.126:35495 shutting down...
I20251027 03:10:42.472658 28253 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 1 LEADER]: Raft consensus shutting down.
I20251027 03:10:42.472716 28253 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 03:10:42.472745 28253 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7e09f0d52e24423897dc971fcbafefd4: stopping tablet replica
I20251027 03:10:42.484194 28253 master.cc:583] Master@127.27.151.126:35495 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (6561 ms)
[----------] 14 tests from AutoRebalancerTest (180539 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (180539 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20251027 03:10:42.490180 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 7 messages since previous log ~3 seconds ago
I20251027 03:10:42.490321 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~3 seconds ago
I20251027 03:10:42.490346 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:4712: suppressed but not reported on 7 messages since previous log ~3 seconds ago
I20251027 03:10:42.490365 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/master/catalog_manager.cc:5327: suppressed but not reported on 21 messages since previous log ~28 seconds ago
I20251027 03:10:42.490396 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~167 seconds ago
I20251027 03:10:42.490422 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 50 messages since previous log ~16 seconds ago
I20251027 03:10:42.490492 28253 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 24 messages since previous log ~4 seconds ago